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Ellis B* Fag^ and Thoman F* Breen III 



(Abstract) 



' Education suffers froB) the lack of any common measure of overall edu- 
cational advancement » related in some clear way to the objectives of so- 
ciety* The present study proposes such a measure in the "bentee" (stand- 
ing for benefit T«score) and provides evidence about its utility and form* 

Creation of a general "bentee"- permits the d^ign of a tree of values 
leading by dearly defined steps ^om the highest values of society to the 
lowest^ m^t direct and definable-objectives and test items. The mathe- 
matics of such a tree is analysed » permitting the tracing of any sub-area 
of performance in terms of its^ overall value* 

'Two methods of assigning values « at a given level of the tree, are 
compared experin^ntally, usind 101 sample judges from two regions of the 
U*S* It was found that the values had clear and consistent rank orders 
when p»oup means were considered; that the less elaborate t^en method 
served as well as the latent technique i that the relative values for boy 
hi^-school seniors were only slightly different from those judged' for . 
girls; and that educators did not differ, as a group, from non-educaitors 
in their assigned values* Individuals, however^ differed enormously Sizm 
one another, in their appraisal of educational values* 




■ Pr«fac« / 

■ ■ i ■ - • ' •• ■■ / 

The idea of th« *'b«nt««" (or of aom« ainilar systes of caabiniog in<* 
fonaation into an ovarall ,'x»ating of educational acconplishiatnt) probably 
haa a long anceatry. . B^t for tho firat author, the beginning notions first 
took form during tha Spri^ug of 1971, no doubt influenced by a good many, ex- 
periences, but notably by these t work with the beta coefficients in multi- 
ple regression; work, in Venezuela f <3r a year on the problem of national 
goals; the chairmanship W ah Applied Research Planning Session for the 
U.S. Office of Education; an earlier concern with educational -philosophy 
as related to behavioral science; and noat importantly, the experience of 
frustration in attes^ting application of management science to education, 
this difficulty resulting from the absence of any general measure of bene- 
fit beyond a single ^est score*' 

The debts are many to the persons who suggested approaches and criti-' 
oized the early thinkings I am indebted to Sariford Temkin for conversa- 
tions in 1967; when he spoke of searching for a unit he called a "bentile," 
which was conceptually to be constructed in a very different way; also for 
guiding me to certain writings such as those by C, West Churchman, and to 
^his own work within Research for Better^hools Philadelphia. For other 
valued comments, I am indebted to Robert Gable, Arthur Jensen, Robert Linn, 
Richard McCann, Richard Otte, Richard Page, Dieter Paulus, Harry -Silberaan , 
and Julian Stanley. Later valuable comments ware' reciev^d from Ca^^l 
Bereiter, Bruce Rogers, and Skip Livingston. Peter I. Tillett made some 
good technical recommendations, «(nd Harry Post en solved a substantial sta- 
tistical problem for us. Important later comments cai^e from William £. 
Coffman, and- from Harold Gulliksen. 

■ ^ / 
The general ideas of tl^e bentee project have b^en encouraged by editors 
anid by program chairmen*. Robert' Linn published the beginning idea in the ^ 
Journal of Educational Measurement (1972). Scott Gehman' accepted a further 
theoretical and mathematical analysis for Educational and Psychological • 

' Measurement (1974). Gilbert Austin arranged an invited address for the " 
American Educational Research Association (1972)) and Lewis C. Solnon, of- 
the National Research Council, encouraged its presentation to a panel at 
Woods Hole concerned with higher education (these proceedings have now been 
published). And the findings and conclusions were pr&vided a very useful 
hearing by William E. Coffman j"^ at the University of Iowa Dedication of ^he 

..Lindouist j;j©Jitfir_JfirLJEiiUc.ati^ pr.o.ceediii.gs. 

have also been published). 

■ The writers are greatly indebted to the officials of the U.S. Office 
of Education, Boston, who provided a small but vital financial support to 
the research. And we thank Wayne Martin, who collected data in the area of 
Cleveland, Ohio. We are grateful to Richard i'^loomer, who helped evaluate 
the proposal and the data, as part of the Disi^ertation Committee for the 
second author. And we » acknowledge the help provided by The University of 
Connecticut Computer Center, with support. from the National Science Founda- 
tion. 

— Ellis B, Page (Storrs, Conn.) and Thomas F. Breen <sao Paulo, Brazil) 
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CHAPTER 1 i 

UBFINITION, BACKGROUND, HYPOTHESES ' ■ . ' 

* - ■ . . * • « 

The purpose of this researtih ijS to, propose and evaluate a measure of 
overall educational accomplishment.' This measure is referred to as.«the 
"bentee" (defined and describe^ later), and may be considered Roughly ana- 
logous to a "dollar" in economic valdes. This chapter wi,ll accomplish' the 
followings Section A will define the problem of overall value. Section B 
wiil examine the related lite.rature. And Section' C viil describe "the, .hy- 
potheses governing this research. ,:' 

, « ■ ;' . ^ ' , 

i A. liiefinition of the Problem 

• """" 

.' Educational ipesearch and development have repeatedly run aground in 
the fog of undefined goals. Long-range human goals, such as "happiness," 
"adjustment," or "equality," seem too remote from curriculum to be useful 
in educational planning. Yet without reference to the largest purposes , 
the sub-goals selected have at best an arbitrary quality: too often they / 
seem to self-serve particular educators or sub-groupa of society, and to 
be lacking in any obvious legitimacy. Educational planners and administra- 
tors are constantly enco\intering competitive goals which are literally "in- 
comparable," since no technique exists for comparing them with each other* 
To 'make -dear this difficulty, let us consider three cases requiring ad- 
ministrative- defeis ions. They are .common problems, and thus exhibit our 
fundamental confusion: \ 

Case iSfl. Recently, in a well-published study, a re- 
' searcher gave low perfowiers remedial work in mathemat- 
ics.'' The students improved, somewhat reducing their 
lag in that one subject. Left unreported, however, was 
what happened to them in other subjects, from which the 
extra time was taken. Lot us assume they suffered slight- 
ly in these others (as would surely be true in a well-run 
school). Then how would one decide whether the combina- 
tion ^ain-and^loss is c'esirable?. 



Case . #2. A school board is considering possible allO" 
catioM of teaching effort. One option is to give heavy 
emphasis to programs for the aclfdemically weak; another is 
to spread available resources more evenly across the board 
of talent I a third is to emphasize the gifted. How may we 
establish cost-effectiveness ratios for the three options, 
in order to compare them? If we define "effectiveness" 
for the ^*eak students, the measure will not work for the 
gifted. How can the benefit to the total youth of the 
district be measured? Surely, the choice should not be 
resolved purely at the level of politics or public rela- 
tions. * " ' 



Case #3. A high school is swept up in a fashionable 
movetaent fop; student "freedowf* in course selection. As 
a result, a large nusiber choose no jnath* no hard science « 
no fsreign language. One body of teachez« argues that 
the students are developing "responsibility" and "ore- 
' ativity," and these values are nore urgent than the purely 
academic considerations. Is there any responsible tech- 
nical procedure for balancing these clains, and for in- 
terrupting the confusing circle of warring opinion? 

Yet this fundamental confusion exists in the last third of the twen- 
tieth century, after five thousand years of educational ekperience« three 
hindred of astonishing scientific development, and more than eighty years, 
of a rapidly improving behavioral science. In the words of pne writer 
(Churchcaan, 1961, p. 1): "Probably the most startling feature of twentieth 
century culture is the fact that we have developed such elaborate ways of 
doing things and at the sane time hav^ developed no way of justifying any 
of the thin^ we do." ^ . \ ■ ■ 

Objective Functions. Vet by coqparison with education, the practi- 
tioners of certain other disciplines have overall goals which serve as, 
at least , an approximate measure of achievement , so that they nsy select 
among competing alternative's: The engineer can use efficiency of manufac- 
ture, measured in cost or number of operations. The industrialist lias 
corporate profit. The physician, client recovery. The businessman, per- 
sonal income after taxes. And for certain purposes, the presence of such 
objective functions becomes a great convenience. 

g ' _ 

None of these objectives' (steps of manufaetui^ , client recovery, in- 
come after ta^xes) is a conpletely adequate representation of the presuised 
total objective. Yet they are very useful as approximation6_emd.£nter 
fruitfully into various formal "games ,^]_t^^^°^4^^T~^nd strategies, such 
as linear prograxmd.n^''%^euQing--analysis, and decision theory (Churchman 
et al, 1957; Hareyd^67'; jEia»^art , 1969; Van Dusseldorp et al , 1971). 
Wagner (19J&S}- Writes that for a problem to be susceptibtle to operations 
research, it must have four characteristics: "a primary focus on decision- 
making ... measurable values [such as economic values] that unequivocally 
reflect the future well-being of the Organisation ... reliance on a formal 
mathematical model ... [and] dependence on an electronic, computer" (pp. 50 
6). All four of (the^e criteria clearly turn around "^Aiq seisond: these 
"measurable values" which will make poss^blo the matheirstical models, will 
justifyothe coo^lexity requiring the cpn^uter, and will validate rational 
decision-m^ing. Lack of an overall e f f ec t i vehe ss crite r ion , therefore, 
makes it ^ery difficult to apply management ^scif^nce techniques in educa- 
tion. / . . ^ • 

Attributes of a Solution . Sv^pose thac a suit«ible measure of educa- 
tional benefit were at hand. Then what tfouXd be its characteidstics? Some 
would be: a universality of application; an interval scale of measurement; 
a non-arbitrary definition; a democratic basis for establishment; provi- 
sion for ej^ert management; responsiveness to appropriate change; and a 
reau;>i>siveness o2 function, so that it might be applied at a number of 



different leyals. Th«se are stated cryptically here but way become clearer 
later in the. presentation. In any case, a model of these qualities is the 
dollar ..(or other monetary unit) , a brilliaht social-invention established 
; a marketplace- of goods and services i . . i " 

Let MS therefore postulate a ''marketplace of educational values,'^ 
•'^What would be the features of such a marketplace?' The principal feature 
Us that many jdif ferent "cases''- should be traded in some useful way , so that 
the appropriate values may be established' on the basjis of one common scale. 
Another is that th-. judgments made should have a potential for traiislation, 
eventually, from value-space to test-space. I 

; . .. • , •-•■*!... 

-\ Over the last several decades, psychologists and educators have be- 
come expert in the' measurement of traits, and only s^ewhat less so in the 
analysis of , ttait ^'profiles" (multi-trait vectors) of students or groups of 
students (Crortbach, 1970, chs. 10, ll)..i' We know how to study relationships 
among traits; yet the separate 'traits still remain separate; ^ we do not 
have accepted techniques for trading "off one ability against another, or 
for deciding easily whether one student (or group of students), is better 
educated or prepared than another4^ Yet historically, progress has been 
made by overriding the\ difficulties and ambiguities in such judgmeKts, and' 
by functioning within ^ome clear reductionist rules, even though recognis- 
ing their limited correspondence with the ideeil* This proposal i,nvest-i- 
gates an attractive" possibility for reducing profiles to single scores., 

M eans and Ends . Since World War II a new management science has - 
emergeTd which l/^is developed techijiques for the analysis of^system^. Its 
main interest is the improvement^f the operations -ofTjafl^machine systems 
through the use of the\ scientif i^ metliod of inquirer Scientific inquiry 
includes '^ogical, systematic,' and empirical anafysis of all constituent 
pa: ts of a system, their functions, and their val\ie in terms of the over- 
all objective. , * . 

Logical analysis begins with the notion of the purposive act itself. 
"Thexje \are three classes of elements, entailed in any purposive act. These 
are (1) ttf!h decision-maker, (2) a set of alternative actions, and (3) a set" 
of goals"" (Churchman, 1961, p. 137), The decision-maker specifies one goal 
from the set of possible goals "and then attempts to x»ptimize the accomplish- 
ment of that goal by selecting from the set of alternative actions those 
which have the greatest probability of bringing him to the goal. If the 
goal h< s not^^been specified exactly enoug^^it«will be impossible tjp choose 
among alternative programs to optimize it. ' ' ^ 

We can speak ok- good and bad\ actions in two senses: a) in an. ethical 
sense, in terms of morally good and. bad— a distinction which is outside the 
present realm of discussion— and b) in a practical, prudential sense, in 
terms of which actions have better or worse chances of accomplishing some 
specified end. A good action is an^ action which is better suit'ed to at.'^ain- 
ing an end. In ethics the designation of a morally good act is predicated 
on an analysis of ultimate ends and goals and of the relationship of human 
acts to those ultimate groals. In practical, prudential judgments, actions 
are designated as goo<^ or bad after an analysis of their chances of accomp- 
lishing their specif l^u purpose. The selection of educational program""^<. 



may involve some ethical consideirations, such aft "the possible, waste of time, 
■energy, resources, and people, but of m.qs?e^ importance to this discussion 
,are practical, considerations of how well various investments of people and 
programs accomplish the ultimate &nd; of ^tducation. - flithojat a definition of 
^the end of education there is no w^ to evaluate ^he means. ' < 

T h e S drools a>». Means . Our schools are the appx^oved and'acce^^^ted method 
of socializing our, young. , Through sciiooling the accouterments of society 
and culture are passed frpm one generation to the next. , It is the official 
mean^ adopted by American and Western European cultures tp prepare younger 
membi^rs to^take their full pl'ace in the culture as adequately functioning: 
^dUIts. "In its very eSsetfce, the, scl.jol is an iTistitution established by. 
.society for the purpose of preg^iiig the young to participate in that so- ^ 
ci^ty'' (St^ley et al., 1956, p. 2).r\ 

' \ This educational system has developed oyer many generations into it^ 
now 'rather complex structure. And its ultim'ate purpose ^may have been lost 
it> tiiat; development « Our scliools. g^t i>igger, our curricula more varied;, we 
employ more advanced technolisgy and- Imacnines , and consume more money and - 
resources in the 'process, '^f^dmf ttedly, numerous substantial, gains have; 
been realized from some of these deyelopjucnts. However,, the question miist 
ultimately be raised: . . .by what standard shall the value. . .of any edu- 
cational ,activity be neasured?" (Thut; 1957, p. 382). 

The Schools have a tendency, as ; does any organization, to trap, people 
in a "mindlessness where they fulfill theii- jobs without considering why 
they are ^oing them, without questioning their jobs' purpose. As Silbisnnan 
(1970) stites: 

i ' . - , ; • 

|. . .by and large, teachers, principals, and super- 
jintendents' are decent, intelligent, and caring people 
■jwho try to do theii' best by their lights. It they 
imake a botch of it , and an ^uncomfortably- large number 
do, it is because it simply never occur3 to more than 
a handful to ask why they are doing— to t^ink seriously . 
or deeply about the purposes or consequencig^ of educa- 
tion (p. U).^ . • \ 

He quotes henry^At Kissinger furthest^ on this problem^of mindlessness : 

. . .d'h increasing amount: of energy has to be devoted . 

to keeping the existing machine going, and'- in the nature 

of tilings there isn^t enough time to inquire into the ^ 

purpose, of these activities • The temptation is great 

to define success "by ."whether one fulfills certain pro- 

graans, however accidentally, these progrants may have been' 

arrived at. \Th3 question is whether it is , possible in 

the moder^n bureaucr^atic state to develop a sense of 

long-range purpose and to inquire into the meaning of 

the activity (p. 20). » ^ 

Without some adequate definition ^f .a goal for the educational sys- 
tem, it is nearly impossible to attempt to justify any of the means 



educators use in.schooXs.l XI is itnpossible, f or the eduaatlonal decision- 
maker to make a rational choice between alternative actions and programs 
without tho:i;Hough definition of the ultimate purpose for which they are 
designed. /The evaluation of alternative programs depends on the relative 
prob'ability that eacb.jgrogram furthers the ultimate goal of the entire sys- 
tem, ithe program ^lilir'tKe' highest probability of success, with the highest 
measure 'Of a^coinprishment ^ being considered best* 

It is striking, theh,^ to realize that education lacks even any coher- 
lent effort to define some Wasuraisle objective. Our schools share some of 
the poor symptoms of other twentie^th century' institutions. c.;Vo .repeat' the 
words of Churchman (IS&l, p. '1): ' "Probably the most startling feature of 
twentieth- century culture is the fact that we have developed such elaborate 
'ways of doing ^things and at the same time have developed no way of justify- 
ing any of the things we do." ' 

A rational choice from alternative programs demands a scientific an* 
alysis of the educational, systemt of it is procedures and decib ion-making 
processes t and of the relationship of all operating parts to the final 
goals , 

Operations Research » There is a science, operations research or sysr 
terns analysis t which contains the tools needed for such an analysis* It > 
is an adaptation of the scientific method %o the operation of human enter*- 
prise • ' *^ ' *^ 

The scientific method has been man's outstanding 
asset in his technological development • Its success 
has been manifest not only In his greater understand- 
ing of the natural world and his ability to predict 
its behavior t but also in. material advances which 
'have lengthened his life, increased his leisure time, 
and enabled him to pursue a far greater variety of . 
^ activities • , . " ■ 

In the^ field of applications man has pragmatically 
accepted theee advances as sufficient evidence of the 
usefulness of the scientific methtdr Yet, perhaps for 
historical reasons, there remained a stubborn resis- 
tance to admitting the scientific method into the mce 
personajL and le^s materialistic affairs of man himself « 

• • eEarly in World War II, the barriers were lifted, 
and the scientific method, after centuries of confine- 
ment to technical problems , was at least called forth 
to struggle with the operations involved in human en- 
terprises (Saaty, 1959, p* !)•* 

'Although Banghart. (1969X admits that the terms "operations research" 
and "systems analysis" are used interchangeably, he does distinguish be- 
tween them: 



Confusion in the literature regarding systems an- 
alysis and qp^rat ions researoh has been oyer Whether 
or not the two phrases are synohynious. In general, '-^ i, ■ * ^ 
this phrase systems analysis is more coropi^ehehsive and 
ioplies a wider look at the problem* The phrase oper- 
ations s^sea^h denotes one of a set of specific mthQ- 
. matical techniques ' fQr problem analysis <p. ^37). 

Definitions. As has beten indicated abov^v^jsystems analysis or opera- 
tions research is the^application of the scientific mgthod^to human enter- 
prises. Most of the 'definitions of this science are couched in geheii^al 
terms. yaaty.(l9$9, p. 3) gives as the "^usual" defiriition..of operations 
research "an aid for the executive in malting his decisicois by providing him 
with the ^eeded quantitative information based on the scientific method of 
analysis."/' y 

. A<^ording to Ban^art (1969), systems analysis Is 

. . '.the set of quantitative-scientific, techniques that 

assists the* . .administrator in the decision-making' 

process. . . .Althot^gh the concept of sys tecs analys.is 

implies an. investigation of a total system, the term < 

denotes, a set of quantitative-scientific tools avail- - 

able for analytical purposes (p. 3). 

. And although Churchman et al. (1957) think that operations research 
"is perhaps too' young to be deefined in any authoritative -way ," the authors 
do provide a "tentative working defij^tion:" 

Operations research is^ the lapplication of scien-. 
tific-jHithods', techniques, and tools to problems in- 
volving the operations of a system so as to provide 
those in control of the system With optimum solutions 
to the problems (p. 18). 

<• 

Because of its great size and conplexityi there is a necessity for 
applying operations research and systems analysis to the educational sys- 
tern* Operations research is a development ainl^d at exercising some con-- 
trol over the coiifiplexities involved in modem systems « Essentially , ^^a 
syst€3m is integrated aajseably of interacting elen^nts , designed to carry 
out cooperatively a predetermined function" (Flagle et^alt t 1960, p. 58)e 
Chorafas (1965) shdws the conib^/nation of the two concepts: 

A system is a grp\^ of interdej^ndent elements 
acting together to accoa^lish a predetermined pur.- 
pose. Systems analysis is an attempt to define the 
most feasible, suitalle, and acceptable means for 
accomplishing a given purpose (p. 2)e 

Whatever scientific and quantitative tools are use'd for research into, 
and analysis of, systems a2:^ sv^bsumsd under the heading of operations re- 
seaz'ch« 



The Sehoel System, W« ganarally speak of the '"school system** or the 
"educaHpnal system," and it can be shown that the schools do form a system 
in the technical sense. They are complex purpciseful orgasiizatioiis* Feyer- 
eisen et -'ai (1970) express this clearly: 

A system is a set of cbs^onents organized in such a 
. wayAto Goastrain action toward the accomplishment of the 
^lorposes for which system exists. Applied to a school 

- organizal^on this moans that a school system is organized 
V. to receive uneducated children as, its input, and through 

a. process called e ducat ien produce educated young adults 
as its output (p. 5£l). 

The size and complexity of the system detatd new, more rigorous methods 
for the development and administration of programs. As Banghart says: 

■ . ,* .the problem has as its focal point the increased 

- size and conplexity of educational organizations. As a 
result of the ^. increased size jand complexity, new tech- 
niques had to be developed to help the administrator with ^ ' 
his administrative decisions (p. 7). 

. However.,. education has. tended to be one of the "more personal and leas 
materialistic affairs of man" which have often shown "a stubbom resistance ^ 
to admitting the scientific method" (Saaty, 1959, p. 1). But the scientist 
must continue to try to m^e available to the educator "the best tools to 
predict the- result's of any course of action he may propose and to analyse 
dispassionately his aine and objectives" (p. 1). 

The historical or psychological reasons why educators may have re- 
sisted opening their affairs to the dispassionate scrutiny of the scien- 
tific method are not to be argued here. What is being pointed out is. that, 
although the schools constitute a system, they have not been operated ac- 
cording to systems concepts. 

. . .Schools exist as systems having interdependent 
^ parts or subsystems. And, as is typical of systems, 
they have inputs, processes, and outputs. They have 
a purpose, a mission, a goal. While schools have 
been called systems in the literature of the past, 
they have not been organized, nor have they operated, 
according to systems concepts (Feyereisen et^ al. , 
1970, p. 7). 

Systems concepts are used to design a self-regu* 
lating and adaptive management system for a school 
which is capable of compensating for the stresses and 
tensions found in the system and redesigning itself 
to ensure .that the objectives of the system are accom- 
plished (pe 55). 



The fGoaoix schools have nat operated according to systems concepts is 
that one of the intrinsic characteristics of a purposeful organization has 
been missing* That, essential element is the control function. According 
to Huysman (1970, p. I'a control function, which compares the achieved 
with the desired outcomes," is an integral part of a purposeful system. 
This function cannot be implemented in education- until the desired outcomes 
are strictly specified. As Page (1972, p. 3U) says, "Lack of an, overall 
effectiveness criterion. i therefore, makes it very difficult to apply manage* 
ment-science/tecimiqueS in "education," 

The purpose of Education . • Therefore, the purpose of our educational 
systemlr^st be specified. The goal in need of specification i§ the overall 
purpos^'of the schools to educate. For Silberman U970, p. 6) "education, 
to be -^education must be purposeful." 
/■ • • , 

/' Education is defined. . .as the deliberate or purpose- 

/ ful creation, evocation, or transmission of knowledge, 

/ abilities, skills, and values. To emphasize 'the 'deliber- 

.'a,te and purposeful is not to deny that non-de^ilBeJ^ate \ 

influences may be more powerful} it is to assert that 
roan cannot depend upon a casual process of learning. 
Unless men are forced to rediscover all knowledge for 
themselves, t;^ey,.mus.t bu educated, which is to say that 
education to be education, must be purposeful (p. 6). 

There has to be some specification of the purposes or mission of the 
jschools befbre operations researcn. and systems analysis techniques can be 
/applied to other than tsiviii...pr^lems. Banghart (1969) is very particular 
about pointing this out: ^ . . ^ 

' ■ » 

'The most difficult part of the entire systems -study 
involves 'establishment of very specific' objectives to 
be accomplished. Because of the quantitative nature 
of systems analysis, it is necessary to be extremely 
specific in detemininc the objectives tp be achieved. . 
To state objectives in general terms such as the im~ 
provement of curriculum, enhancement of the educational 
system, or development, of a better building program is 
totally inadequat^\(p. 39) • ^ ^ , 

An. evaluation of performance is done in terms oft ef- 
fectiveness. The design of the problem solution deter- 
mines how effective the new system will be in fulfilling 
the organization's mission. It is necessary to devise 
' * standards for measuring effectiveness (p. ^2).. j 

' Inherent in establishing the pe^:formance crit^iria is 
the* concept of validity. Validity is used here in the 
statistical sen'Se as in psychological testing. That is, 
the procedure or performance is valid if it does what it 
is supposed to do. Therefore, the problem solution is 
valid if thi^ solution accomplished what it is supposed to 
accomplish (p. 42). 



If operations research methods <ire effectively applied the specifica- 
tion of objectives will of necessity be of the overall and most general 
objectives of the^system* What is needed is some measure of what it would 
mean to be educated. For, as Churchman at al . (1957), say, "operations re- 
search tries to find the best decisions relative to as large a portion of 
a total organization as possible" (p. 6), It is "not effectively used if 
it is restricted to one-fUot projects" (p. 8^. They cojffninues 

. . The concern of operations -.research with finSikig an 

optimum decisions poiAcy» design is one of its es- 
'sential cliaracteristics. 'It does not seek merely to 
find a better solution to a problem than the one in 
use; it seeks the best solution. It may not always 
find it because of limitations imposed by the present 
state of science or by lack of time, funds, or oppor- 
tunity. But operation research's efforts are contin- 
ually directed to getting the optimum or as close 
to'it as possible (p. 8). |- 

The present limitation in the schools is the lack of a specified measurable 
pverall oi)-jective' for education.. Van Dusseldorf et al. (1971) show some 
applications of operations research- to subproblems within the educational 
system, but an optimal solution, to the entire sysftem, demands, a specified 
overcdl objective. \ 

A Possible Solution . While it may be very difficult to formulate this 
objective, Feyereisegi et al. (1970) do npt view construction of specific 
objectives as impossible. . i 

The goals of an organization express its reason for 
being as well as its aspirations. They project expec- 
tations and define products of the enter^jrise. Educa- 
tional' organizations define their functions in terms 
• of educational outcomes and in language which connotes 
the development of persons in the acquisition of such 
skills and abilities, concepts and perceptions, atti- 
tudes and values, and knowledge and methods of inquiry 
as are necessa^ to personal fulfillment and societal 
improvement. Four principal sources of objectives are 
generally identified as (l)^the learners— their needs, 
interests, and purposes i (2) society— local, state, 
national, and world communities; (3) human activities 
and social processes; and (4) the organized bodies of 
knowledge or disciplines eiobodying the accumulated ex- 
^ periences of mankind. For each of 'these referents it 
is possible to -articulate specific objectives in terms 
of tne outcomes of learning, at any level ^of educational 
effort Xpp. U7-118), 

Once specific objectives have been articulated, what would .rejpain 
would be to combine them, through appropriate weighting techniques, into- 
the overall measure that is needed to develop opti^ium decisions about 
profprams and practices. Such a measure might have the following attributes 
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I ■ . 

which Page (1972) and Page and Breen (1972) consider essential to a suit- 
able measure of educational benefit. They would be: ' 

\ ' ■ ■ 

a universality of application; an interval. scale of 
measuremonti a non-arbitrairy definition; a democratic 
basis for establishment; provision for expert manage- 
ment; responsiveness to appropriate' change; and a re- 
cursiveness of funnt^n, so that it might be applied 
at a number of different levels (Pager 1972, p, 34), 

,With the developmenc of such a measure, it would be possible to take 
a more 'systematic and scientific approach 'to program planning and develop- 
\ ment. It would become possible to assess more rigorously the overall efr 
Nfects of curriculum changes and to calculate returns for* various investments 
in people, money, time, and resources. .Such scientific analysis xi impos- 
sible in the present state of undefined educational go^s'. « 7 

„ Section B w.ill examine some of the goals and purposes stated for edu- 
ction. . Not one, however, is found that fulfills all the needs of the ef- 
fef tiyeness criterion, that is necessary for the introduction of systems 
metliodology into curriculum planning. That criterion must .be, developed. 

- ■■ • / 

• • B. Related Literature 

The above section trijed to establish the need for an overall measure 
of educational advancement to serve as an effectiveness criterion for edu- 
cational programs and planning. Advancements in management science and sys- 
tems analysis, which liave been so effectively applied in ot.ier complex op- 
erational systems, will find very limited application in education without 
some overall effectiveness criterion. A search of the literature will de- 
termine whether such measures now exist, or whether a methodology need be 
developed to produce, it. 

Already expr^essed is a desire for some measure which would have many 
[ of the qualities of the dollar, such as interval scale of measurement and 
a value established in a marketplace of goods 'and services. A theoretical 
solution to this problem would be to turn out different types of school 
graduates into the social marketplace to de,termine what price will be paid 
for each one. Tram this theoretical background there has energed one 
widely used measure of educational benefit and, in terms of its usefulness ' 
for research, it has been highly successf-ul. It is a consideration of 
schooling's impact on a student's wage-earning ability. 

Human Capital . As it turns 9Ut, the researchers who follow this -rea- 
soning have used a measure which Tftot only has-'many qualities of the dollar, 
but is, in fact, the dollar itself. The investigators, many of whom are 
economists, have restricted their research to the economic effects of edu- 
cation and have adopted "personal income as the criterion variable. This 
is the basic idea behind the concept of "human capital," a notion that 
society rewards its better-educated and better-skilled members with in- 
creased econcmic benefits." Becker (1964) points to the 



11 



N tremeAdous amount of circumst^jitial evidence testi- 
fying to the economic iitportance of human capital, 
esj^ecially in education. Probably the most impres- 
sive place of evidence is that more hi^ly educated • 
and skilled persras almost always tend to earn more - 
\ than others (p. 2). . ° 

Within the human capital perspective", people invest ip, further educa- 
tiwn and s^ill improvement to increase their economic welfa^. According 
to Becker knd Chiswick (1966), 

. * ■ . . 

Each person is assumed in effect to maximize his 
economic welfare by investing an appropriate ^amount 
in human capital, and the distribution of earnings 
is determined by the distribution of investments, and 
the rates of retum (p. 368). 

This is the underlying notion of an earnings function, that earnings are 
dependent on schooling and .education. A basic statensnt of tn)^ function 
can be found in Hanoch (1967); 

The flow of net earnings (y) that an individual 
e sheets to receive at a given time is assumed to' be 
a function of his age (t), his schooling level (s), 
and various additional factors, lumped together. . . 
in a vector of variat>les (z) (p. J310). 



At; first, schooling level was measured solely in terms of years of 
schooling since thbse were the data most eeisily available and since they 
did show a signifi^nt relatipnshipHo earaings. 




Limitations of the data available ha^e reduced 
the scope ^f the enpirical^ analysis to investnent 
ii\ formal education as^m@asurable by years of sciiool- 
ing. Evidence from states and regions witlujs^he 
United Stf^es-^and-fpom-se^ral C9untrl?es<~i^i44ca^s 
that sSh^ling- usually explains a not negligible\ 
part of the inequality In earnings within a geo- 
graphical area and a mudji larger piurt of the dif- 
ferences in inequality between areas. These and , 
other findings are generally quite consistent with 
the implications of the tneory (becker and Chiswick, 
1966, p. 368) 1^ ■> ^ 

However, further research has refined some of the measures of school- 
ing level s"© that it became "important to know how much people learned 
while in school" (Hansen et^ al . , ly70, p. U09). These authors say that it 

well known that the level of educational attainment 
(LEA) is positively correlated with level of earn- , 
• ' ings# . .It is erroneous to attribute all ot the ob- 

served differentials in earning associated with 
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ertucation to 4ifferences in years of schooling ^p. 

The significance of years of schooling ger se^ di- 
minishes with iniB introduction of a measure of learn- 
ing. . . .A more fundamental approach. . .would ex- 
amine the factors affecting learning. . i.e. , 
afiility, motivation, home environment and quality of 
schooling, to determine their individual and collec- 
tives rc.les ^p. »*17). 

All of this comhines to help form the notion that schools should be 
the instrument to re.ij.eve the economic inequalities in thu nation." This 
idea has been exported to other countries also, whertf the building of a 
school is seen as another istep toward economic development and social equcuL-^ 
ity. In this country the idea has been tested in a number of major studies, 
ciaminating in the hfghly publicized an^ controversial work on I nequality 
by Jencks ^t al, (1972). " 

In the popular ijiind this work has labeli^ schools 'as failures: 

» 

The public schools are failing dismalXy in what ' . 

has always been regarded as one of their primary 
tasks— in Horace Mann's phrase, to be "the ^at 
equalizer of the conditions of men" facilitating 
" s.\ the . movement of the poor and disadvantaged into the 
mains^tream of American economic and social life" 
(Silberman, 1970, p. 53).' 

Limitations of Human Capital. The human capital line of reasoning has 
proved //ery successful in methodological terras.. The criterion has been an 
objectively maaaured, universally meaningful variable— dollars and cents. 
Criticisms about methodology have" been leveled at many studieis. Levin 
C1972), for example, strongly faults jJencks et^ al . (1972) for their com- 
putation of variables and dismissal <^ certain effects as "negligible." 

Yet more inportant than the computation of variables are the philo- 
sophic limitations of the method itself. There is-, first of all, the p<»- 
sibility that the economic effects of education are only part of its total 
aim, and ppssibly only a- minor part. Becker (1964) had some re- 
strained but harsh words to say about such critics: 

Passions az^ easily aroused on this subject and 
even people who are generally in favor of education, 
medical care, and the like often dislike the -phrase 
. "human capital" and still more any emphasis on its 
economic effects. They are o^en the people who 
launch the most bitter attacks" on research on human 
' capital, partly because they fea:^ that emphasis of 
the 'material' effects of human capital detracts from 
its 'cultural' effects, which to them are more im- 
portant (p. 2). ' ' ' ^ , 
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Thus, there are probably^ many people 1^0 would not want to see courses in 
the fine arts offered only to budding artists^ or physical education courses 
only to prospeeicive professional athletes. The effect of the human capital 
view is to reduce cultural values to a question of economic profit* 

Ano'ther lioltation, pointed out by Schultz (1967), is that there is 
"all too little evidence on the relationship between social and private 
rates of return" (p. 308). When hunan resources are considered in economic 
terms only, they are not measured in terms of their value to society in gen- 
eral but only in ^erns of their value to individuals. It is, in fact, im- 
possible to piaffe all menbers of society in the highest paying positions, 

Altena^ative Goals. Not all ^iducators are ready to accept the economic 
^analysi|/of. human capital as the prime measure of the benefits to be derived 
fro^^r^cation. Although easily measured, it is not as universally accepted 
a^<^ overall measure of educational benefit should be. Jencks himself 
Admits in an interview (Robinson, 1972) that there are other important oxzt- 
comas of schooling beside income. 

. . .moral develqpms^^t. . .seems to me a rather 'im- 
portant issue, to say the^east. . And we don't knew 
anything about whethe:i^ schools have any effect on 
this. . . .There's a long list of things we. haven't, 
measured. So we can't say what kind of impact school-^, 
ing makes in many areas of life (p. 257). 

. *The famous Coleman Report (Coleman et^ al. , 19615; Hosteller and Moyni- 
han, 1972a) assessed educational opportunity in terms of academic achieve- 
ment, the results of tests of academic achievemsnt. This, again, is only 
one kind of impact schooling makes. As Mosteller and Moynihah (1972b) 
point out J- --' 

«■ ■ ■ 

academic achievement is but one output ,' and. . . 
schooling is e}q>ectcd to produce many others. Re- 
tention rates , proportion going to college , income 
and occupation of graduates, even happiness, are a 
few of many outfputs that might be measured (p. 6). 

The three R's do hot end the li-.t of things we 
want schools to give our children. (Surely , for ex- 
an^le , we want children to emerge from thiir school 
years with a sense of social dignity and place, and 
with a commitment to their community. And we want 
them to learo how to work and live with others) (p. 
27). . 

At this, point a synopsis will be presented of some of the purposes of 
education expressed by educational philosophers, thinkers, mid critics. 
One cannot guarantee that all voices will be presented. It is also impos- 
sible to point out all the deficiencies in each case which make it imprac- 
tical to apply operations research techniques to education. It should be 
sufficient, however, to see that there is general disagreement among the 
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many purposes. Some of these are disagreements of emphasis, but many are 
fundamental- differences about the nature of education itself. 

♦ • * - t 

For what an individual values in education may depend to a great extent 
on both his experiences with education and his present relationship' to it. 
For example, Feyereisen et (1970) feel that "a statement of objectives 
prepared by academicians ..will usually be subject-matter oriented'* (p.' 138). 
It may be possible to ^ee such^ relationships in the objectives presented ■ 
here. • ■ ' . 

S pecified O bject ive 3* Since our discussion is particularly about the 
ends of education in the United States, which may be discinct from other 
cultures, this survey begins with the great Ameriqan philosopher, John 
Dewey, who emphasized in Moral Principles of^ Mucation, that "the moral pur-- 
pose is universal and dominant in all instruction — whatsoever the topic" 
(quoted in Silbesftan,- 1970, p. 9). 

Not ail peop^b^^would /agree . For Ebel (1972), moral 'education deserves 
high priority, "but^tt^Uoes not deserve the highest priority. That spot 
.must be reserved for the ^cultivation of cognitive competence" (p. 5). He 
feels that education must noove away from trying to solve .the problems of 
young people to imparting useful knowledge/ ' ' * 

Schools have been far too willing to accept rer- 
sponsibility for solving all the problems of young 
people, for meeting ail of their immediate needs. 
That schools have failed to dischar;ge these obliga- 
tions successfully is clearly evident 

. * ^ ' 

Schools are for learning. They 'should bend most 
of their efforts to the f^ilitation of learning. 

The kind of learning on Which schools should^ con- 
centrate most of their efforts is cognitive compe- 
tence, the command of useful knowledge (p. 7). I 

Rogers (1969) is in general agreement with the term "facilitation of 
learning»*^ but makes a definite qualification: 

teaching as imparting of knowledge makes sense in 
an unchanging environment;. We face a new situation. » 
The goal of education is jthe fjacllita^^^ of chouige^ 
and learni ng. The educated man is he who has learned 
how to leaVn; how to adapj (p. 104), * ' 

Whitehead (1929) protests against dead knowieage and inert ideas. 
'^Except at rare intervals of intellectual ferment, education in the past ^ 
has been radically infected with inert ideas" (p. 15). Education should 
not be the imparting of knowledge as such, but of the ability to use it. 
''Eaucation is the acquisition of the art of the utilization of knowledge** 
(p. 18, quoted in Stone and Schneider, 1965, p. 30). 
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In this regard he is in some agreement with Individuals and organiza- 
tions who. think that education should have a great deal to 'do .with the 
ability to think. According to the Educational Policies Commission of the 
National Education Association <1961)« '*'the purpose which runs through and 
strengthens all other educational purposes— the common thread of education- 
is the ' development of th3 ability to think" (p.' 12). And Ennis (1962, p. 
82) writes that "the ^.teaching of critical thinking. • . is thought to°be 
one of the educator's main jobs." 

T 

But for. others t such as Pritzkau (1970) ^ education is a* form of gen<* 
eral self •realization and transformation* Ke speaks of education for the 
authentic,'^ calling for the development of authentic people through seme 
encounter t facilitated by the teacher t with the authentic meanings of people 
and things 

And for Freire (1970)^ fbom his experience in northeastern Brazili its 
poverty and social ferment ^ education should be an instrumep/c for the trans- 
formation of society » for soQial and cultural revolution«~^ducation should 
be for the liberation and humanization of people t fpr training their con- 
sciences to see the evils and oppression in society and to want to do some- 
thing about it* ; ^ 

Finally I education is seen as an instrument for teaching people how j 
to live 9 which may be a return to Dewey ^s ideas* According to Howe (1963)1: 

The purpose of education is not simply the trans- 
mission of information but the* bringing into being of 
persons of respoz^ibility and integrity who /in a world 
of persons and things, can be'. instruments of love which i 
in the words of Paul Tillich, *moves everything toward 
everything else that is**' The .educator *s responsibility, 
is to recognize that each single unique person is the 
bearer of a special task of being which can be fulfilled 
through him and him alone (p* 70)* 

The above selection of views on the purpose of education shows that 
the unity of thought which would be desirous on a topic of such vital con- 
cern to educators is certainly lacking* A further deficiency is th6 im- 
mjeasur ability of many of the goals expressed* And thirdly i some statements 
lack specificity of the purpose, i*e*| what types 6f activitieis are ex- 
pected to achieve the goal* , 

There have been some efforts to develop a systematic analysis of edu- 
cation and its purposes* One of them originates from the Commission on the 
Reorganization of Secondary Education in 1918 which decided on seven cardi- 
nal objectives: health; command of fundamental processes; worthy home mem- 
bership; vocational^ competence; effective citizenship; worthy use of iLeisur^ 
time; and ethical charactar (Stone and Schneider, 1965, pp* 27-28 )« An- 
other list of purposes wiS developed in 1939 by the Educational Policies 
Commission of the NEA, ^nentioned above. It summarizes objectives under 
four main headings: the objectives of self-realization; the objectives of 
hUftiaii relationships; the objectives of econcmic efficiency; and the objec- 
tives of civic responsibility (EPA, 1961, pp. 31-33)t 



The Question of Valu e > Yet even when one has systematized/ and speci-* 
fied all the objectiveai oT education! there retnuins the question of Uow 
valuable each objectiv€i is in itself and in relati,pn to other objectives. * 
This brings^ us to the study of values, xnavm_a_s_juci6iogy. 

As a rule, a study of values centers around three 
main issues: (1) whether values are objectives or 
subjective, that is, imperso&'al or personal; (2) 
whether they are changing or constant; and (3) whether 
there are hierarchies of value. 

Objective values exist regardless of man's per-* 
sonal feelings and desires. • • .Subjective values, 
on the other hand, are relative to personal desire, 
which confers value on the object considered. . • . 
Absolute values are constant. . .and are a reflection 
' of reality itself. . . .Changing values,. on the con« 
trary, are likely to be xjespouses to man's immediate 
needs (Knellert 196H, pp/ 13-14)^. 

Brubacher (1962) makes this comment about values; . 

^He must distinguish what in fact 13 held valuable from 
*what ought to be held valuable. While we know pretty 
well how to settle disagreements about facts of value — 
for. they do not differ greatly from other kinds of 
facts«»«^we find it much more difficult to settle di^r 
put^s about what ought to be the fact or norm of 
va^es (p. 110). ^ 

The myriad goals proposed for education give a hint cf the difficulty 
of settling disputes in the context of what, ought to be the norm of^^^alues. 
It makes s us despair of finding, through the- exercise of intellect alone, 
some "correct" objective function which would illumine the intrinsic merits 
of all educational goals. ' It may not be desirable or even necessary to . 
attempt that. 1 

We can call education "the process whereby persons intentionally guide 
the development of persons" (Phejf^ix, 1958, p« 13). The general purpose of 
this development is to prepare ^dults who can function adequately in the 
society. According to McCleary and Hencley (1965), this preparation, to 
exist as formal education, majf convey 

at least a modicum of each 'of the following elements: 
(1) a development of the individuai"personal, emo- 
.tional, and aesthetic aspects of life; (2) modes of 
thought and techniques of inquiry— disciplined intel-* 
lect; (3) transmission of the cultural heritage — civic 
and nu^ral values; and (H) promotion of vital societal 
needs 'contributing to econcmic, social, and political 
well-being— the realm of technics (p. 195) • 
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Yet the relative value of. each of the elements, and of their constitu- 
ent parts , will depend very much on the nature of the society iiito .which 
one is to be assimilated* ^ * . .. j ' 

The ultimate aim pf education, therefore, takes it4 
fons from whatever the social stereotype may be. Natu- 
rally this stereotype will vary depending on whether | 
society is organized along ('lemocratic, fascistio, or '{ 
communistic lines^^(^;»Jbacher , 1962, p. 115). ' j 

"Therefore," ^according to Feyereisen ]^ al. (1970, p. 138), "we 'must start 
be defining the needs of the 'individual, the nature and needs of society, 
and a system of values from which we can derive the objectives ^qf the ■ 
curriculum," . * 

Most importantly and'finalJ^, it is from the needs of society and its 
system of values that any justifiable measurable objective should be de- 
veloped. „ 

« 

In its very essence ^ the school is an institution 
established by society for the purpose of preparing 
the young to participate in that society. Like the^ 
family, the church^j or the government » the school is 
a social institution whose fundamental character is 
determined by the society it serves ( Stanley t -et 
1957, p. 2). ^ 

' Although the instrumentation for accomplishing this governance are not 
developed for^ maximum efficiency, society does rule the schools^*' ' 

Vmen^ the error between what the p^ents want their 
children to learn in school and what the children actu« 
ally^ receive be^cones too great, the school board feels 
this force of px;Ublic ^opinion and changes the school poli-- 
ciese In suc^ a manner, the schools are regulated by 
the mores of the community (Smith, 1958, po'3)» 

Solution. A solution would lie in developing objectives out of a study 
of the values of society as a whole« ^ Jensen (1971) investigated educational 
values. However, his study dealt with beliefs about .the process and methods 
of education, and not about the products or objectives. More needed is a 
nieasure of the judged value of educational objectives. ' 

Each objective of the curriculum should be measured against society ^s 
values. Some overall scale should be developed which incorporates each 
aspect with its individual relative worth* Then this scale caa be used to 
measure the accomplishment of every schooled member of the society. It can 
be calculated periodically to adjust appropriately to any changes in the 
society* It can be used to measure the effectiveness of the school by its 
ability to have its. people measure up to the standard. And the procedure 
of comparing objectives to values can be applied at all levels of the cur- 
riculum, if it .is found to be organized hierarchically* 



These are, in fact, the attributes which Page (4.972) and Page and Breeh 
,(1972) propose fo^ the acceptahle solution. ' w 

■ ' * * 

Churdiman et^^l. (1957) have sugg^isted a method of' assigning relative 
weights to alternative objectives , which is to present interested {People 
with possible outcrxaes, or sets of possible outcomes, and have them select 
the most desirable one ^on each set* The most important outcome is arbi- 
trarily ^signed a veUiue of 1.00 and other outcomes are scaled relative to 
it. Coraninations of the outcomes are then compared, on' the assxanption that 
values are additive, and the relative weights refined. With this methoc it 
'W^J.d be nnssible to scale all' the objectives along one continuum. 

In attempting to measure any value-system it might be more reliable to 
record peoplu Is"^ responses in life-lik^ situations than to elicit from them 
self-conscious statements of their values. As Raths ^tay (1966) writes: 

/• . .because values are a part- of living, they oper- 
ate in very coctp lex circumstances and usually involve 
more 'than sin^ple extremes of right and wrong, good 
and b^, true or false. The conditions tinder which 
behavior, is guided, ^n which values work, typically 
involve conflicting demands, a weighing and a. balanc- 
ing » and finally, an action that reflects a multitude 
of forces. Thus values seldom function in a pure and 
-abstract form. Complicated judgments are involved 
and what is really valued is reflected in the outcome 
of life as it is finally lived (p. 27). 

In the context of the importance of measuring values in lifelike situ- 
ations, Page (1972) suggests that a measurable overall educational objep;*.^ — 
tive be developed by an empirical strategy. He originally proposed Are- - 
senting to a random sample of people from the genoval pdpulation, or from 
any other chosen set of "judges,** a number of hypothetical students. The 
attributes of these students would be presented in a profile format as 
scores' on various "academic" traits. These traits can be structurally 
linked to all the various aspects of the high school curriculum. Page pro- 
posed calling this measure the "bentee," for benefit T-score, a normalized 
measure of the benefit derived from education. 

Method . Accordingly, one method of , this study was^ to abstract from 
the lifelike situation those elements of possible student-graduates, out- 
comes of the system, which are felt to relate to a schooling experience. 
A group of "judges," will be asked to evaluate these outcor.es. The solu- . 
tion to the problem of defining and weighting educational objectives will 
be left to an analysis of the judgments of the evaluators about the hypo- 
tnetical students. Possible limitations to this solution lie in the quality 
of the abstraction of student characteristics and in the quality of the se- 
lection of judges. , 

^' » 

Possible outcomes of the school system, which will be students with 
various trait profiles, will be presented to a sample of randomly selected 
people. These people, called evaluators, are asked to rate th§ students 
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on a one-dimensional continuum according to how "well-educated" or how 
"well-prepared" the students seem to be. The value or weight assigned by 
the evaluator to each .trait is the correlation between the trait scores and 
his overall evaluations. 

The study is cpncerned with any systematic relationships between the ^ 
trait scores and the evaiuators* judgments, whether these relationships 
differ for different identifiable groups, and whether the relative iriipor- 
tance of traits are related to judge variables. 

^ A more direct value measure is also obtained from the judges—a weight 
ing of the traits by direct assignment of tokens. This second weighting of 
the traits may be compared with the trait -value measures derived from the 
analysis of the profile evaluations. 



C. Hypotheses 

Hypotheses . The najor hypotheses investigated, as expressed in null 
form, are the following: > ^ 

1) that there c^e no differences in the relative importance assigned 
to the traits as indicated in the evaluators' ratings of the profiles, i.e. 
that the correlations between trait scores and the ratings are not signifi- 
cantly different from zero: ^ * ^ 

2) that there are^no differences in the numbers of tokens assigned to 
each trait by the evaluators \ " ^ 

nt^ - ntj = . . . = nt^ = 50/7. 

3^ that there is no relation between the weightings obtained from the 
correlation study of the profiles and the assignment of the tokens, that 
the rating of .profiles is not a reliable measure of educational values; 

(for each judge and group of judges) 

r, .-^ 0, ^ . 

4) ,that the value profiles obtained for different strata and differ- 
ent identifiable groups are not sip,nif icantly different; 

. = r. . P =^ professionals 

i' nt . = ntj^. j = 1 to n traits (7), and 



5) that there is no significant relationship between any persoral 
variables and the z^elative weightings assigned to each trait a 
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It was predicted that the first three null hypotheses would be rejected 
at a suitable level of significance (p < .OS). The last twb^hypotheses may 
be accepted in most instances. Appropriate statistics > correl at iion- proce- 
dures and analyses of tvariarice » were applied to the data to test these hy** 
pothesesr . ■ • ' 

* 

Finally, a factor analysis is performed on the weights cieri^ved from 
both the direct Ctbken) method and latent (profile-rating) method. This 
analysis attempts to ascertain whether in We view of the ''judges" the 
traits seem to fall into groups, whether the '^judges" tended to treat, 
traits in subgroups and evaluate "students'* on these subgroups rather than 
on the individual traits. 

Summary , A number of proposed objectives and traditional criteria was 
examined. None was found acceptable as ,a suitable, comprehensive objective 
for operations research and systems analysis. The objectives are too nar- 
row in scope, too^ devoid of specification, or too difficult to measure. ^ 

The concept of "human capital" was discarded as. a valid overall criter- 
ion for educational effects, because it too narrowly limits the work of the 
school. The same is true of one-subject academic achievement tests. 

Many proposed goals for education in the literature are found to dis- 
' agree with one another, to be not measurable, and to consist of diversified 
objectives whose relationships aire not defined, or which reflect only per- 
sonal and individualistic views. 

The lack of a suitable objective emphasises the need for measuring the 
cultural values of society toward the traits of possible outputs of the 
school system, since it is to society and its values that schools musjt jus- 
xify their work. The measurement of values should bccur in a lifelike sit- 
J , uation so that the actual values which govern society ''s everyday .act ions 
will emerge. 

A method is outlined defining an overall criterion for education,^ 
through the collection of soci^il values. • Hypotheses have been developed 
about the results of testing this methodology. Subsequent chapters will 
outline the characteristics of a "tree" of values, will speciiy procedures 
of data collection, will analyze "the results of . such data, and will suggest 
further desirable research. 

c 

f 
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CHAPTER 2 
CHARACTLRISTICS OF THE BLNTEE 



The. first chapter describ^dth^ problem of educational objectives in 
terms of philosophical analysis T^-s^-i^^pos^^ tihe ^^bentee*' as an overall 
measure of ef^fectiveness • The present chapter addresses itself to the psy- 
chometric history of .such overall measures, and to some mathematical prop- 
erties of the .proposed tree of values. 



* A, Historical Perspective o_f the Bentee 

» » 
This is not the first time , of course i that someone has suggested some 
sort of "overall" index to summarize a profile of scores (for an early scale 
of "attainment", see jSreene ejt al., c, 1933), An "EQ" was createa long ago 
by analogy with the IQ: by dividing the '^educational age," however , .com- 
puted, -^by the chronological age. And the "AQ" was invented as an "accom- 
plishment quotient," 'computed- by dividing 'the EQ by the IQ, and thus, in 
theory, usable to identify over- and under-achievers (Hilden and Skeels, 
1935). The AQ , howevez^, suffered from serious measurement shortcomings, 
quite apart from 'the scaling' feasibility of the EQ'CCureton, 1937 j Flanagan, 
1951 i Angoff, 1971). 

In more recent decades, there have been corporate efforts to' resolve 
achievement profiles into some acceptable summary scores, in testing pro^^ 
grams of major companies. The National Merit Sdiolarship Qualifying Test, ^ 
in order to make possible a;Selection among students, resolves a profile in- 
to a "total" score. The Metropolitan Achievement Tests summarize sub- 
measurements of usage, punctuation, and capitalization , into a "total" 
score for languageV The Secondary School Admission Tests assemble, from 
part-scores, an "ability total" and a "reading total." The National Edu- ^ 
cat ional Development Tests summarize five sub-scores into a "total." The 
California Achievement Tests advertise a reading total, an arithmetic total, 
a language total, and just a "total." ' Ana the American College Testing 
Program uses five content fields to generate a "composite." The weighting 
rules by wh'.ch ^such composite summaries are developed, however, are quite 
capricious and obscure. 

Seen analytically, all test scores are somewhat mysterious or insecure 
in their weighting rules.. The most professional andv^expert test-aevelop- 
ment techniques now operating", may be r&soiX'e.d into q\iite arbitrary indi- 
vidual decisions concerning ite^m-inclusiou and hence, the implicit weight- 
ing of sub-traits into larger traits. The standard quality control tech- 
niques in which the profession puts such faith (such as indices of diffi- 
culty, discrimination, predictive vaiiuity) still ao not remove the arbi- 
trary micro-trait weighting built into achievement test scores at "almost 
every level. For they do not explicitly come to grips with the values of 
society.' 

Such weighting as has been aone depends ordinarily ou the implicit 
judgments maae by a small number of involved professionals, on the 



availability of items, on'.estimi^tes of predictive validity, on the problems 
of pleasing prospective customers, etc. A distinguishing feature jaf the 
bentee is that weighting is performed by defined, analyzable, and selected^ 
populations of experts and/or laymen whose values may be considered bjr, clefi- 
nition appropriate within a given society. 

The more technical efforts to assess values have usually concentrated 
on individuals, 'and the maximizing of individual satisfactions measured in 
"utiles" or similar scales (for ways" of using values in formal decision- 
analysis, see Raiffa, 1968 j and for- theoretical discussion of the individu- 
al/ group" problem see Churchman, 1961, ch,. 12), Yet in Western education, r 
we usually recognize that the larger pedagogical values or objectives, 
whether ve"t»bal or behavioral, whether norm -referenced or criterion- refer-^-*'' 
enced, should originate in the larger society, or should at _$M_very leasp--^ 
operate with that society's consent. But the ways of translating societal 
values into educaional actions have not been technically defined. ^ The 
bentee therefore represents an effort toward such technical de'finition and 
translation. 

B • ' The Bentee and Behaviora l Objectives 

The movement in favor of '^behavioral objectives" seems virtually to 
have monopolized the scientific study of educational punpose. Yet begin- 
ning with such detailed objectives may lead io overwhelming difficulties; • 
such a "bottom-up" strategy, appears extremely cumbex^some to apply, generat- 
ing huge unmanageable *lists of behaviors , with no clearly understood tech- 
niques for weighting them or eliminating them from the lists. On the other 
hand, practicing educators have usually establishea arb/trary, large, re- 
quired blocks of stuc^y (such as "three years of high school math, one hour 
each day".), which se^m of doubtful authority and genealogy. Also, sucn 
blocks do not lead^^^ any overall measure of educational benefit, nor con- 
tain any clear technique for evaluation of outcome. 

Consequently educational research, development, ana evaluation have 
repeatedly run aground in this "fundamental confusion" of undefined goals, 
explored in the first chapters In the present confusion, there is no- 
accepted technical procedure for trading off, for example, a gain of math 
against^ loss in English; of balancing the claims of one group against the 
neeas of the majority i of evaluating new trends setters who claim one di- 
mension of growth to be more important than others. This fundamental con- 
fusion, furthermore, nfetk^s education inaccessible to management-science 
techniques, which require^^measurabie values • • . that unequivocally re- 
fAect the future well-being of :the organization" (Wagner, 1969, p. 5). For 
such a "measurable value"' would require that any benefits be appraisea 
along the same dimension, with the same scale, the same unit. In economics ^ 



The convergence of group opinions toward a statistical "consensus" has been 
studied in earlier experiments in social conformity, and more recently 
through the "Delphi" technique (e.g., Daikey, 1969). Such exploration of 
belief shift under group influence may eventually have relevance for under-- 
standing the Lientee or similar applied techniques. 

> 
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we have such a unit, in the dollar, franc, or similar monitary measure. In 
education, however," as we have seen, no such measure seems to exist, ' 

i'or these reasons, as already outlined, such a measure of "general 
educational advancement" is,,proposed, to be called the ."bentee" : a coined 
term referring to a "benefit T-score" for education, that is, a- normalized i 
equal-interval scale, adjusted to a norm of some comparis^orT' group (such as • 
high-school seniors, .>or any other reasonable population). -It would have a 
mean of 50, and a stahjLard deviation of lo; ana therefore a usual range from- 
around 20_to around 80 A bentee score, then, is an iaeal, overall appraisal 
of an individual, or of some sub-group, compared with that norm population. 
A person who received a bentee of 70 would be considered "very well -educated" 
or "-prepared". Or a group of students with a bentee score of i+O would« be , 
on the. average, one standard divi'ation below the larger norm population in 
"education" or in "preparation". 



C. The Tree of Values 

Before we cpnsider ways to establish such a measure, let tis consider 
the general shape^of a, top-down value tree. Figure 1 contains a hypotheti- 
cal tree, extending from the highest value- space, almost to the specific . 



Bentee 




Iambic 
Pentameter 



Figure 1. The Possible Recursive Feature of the Bentee 
Technique. (As analysis moves from the general to the 
specific, a shift is made from societal to expert opin- 
ion, and from value-space to test-space.) 
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it«ro-rspace. We begin by arbitrarily declaring the ben tee as the overall 
\nit of value, worth the tptal unity of value, or 1.00. Next, we declare 
t:hat seven "traits" may be' defined such that they include all important edu- 
cational values. In this> illustration, these seven traits are declared to 
be VERBAL, QUANTITATIVE, SOCIAL STUDIES, NATURAL SCIENCES, ARTS, PHYSICAL 
and PERSONALITY. (See descriptions of traits in Appendix). Each of these 
terms becomes a node in the tree, and each node .has its own branches , lead- 
ing to other nodes. These nodes are here presented ^bitrarily ,-but they 
might be matters for articulation of expert panels, regarding the number of 
branches beneath each node, the terms which make up the subdivision into 
branches, and the definition for each node. 

" " ~ (I 

In Figure 1, we observe one possible division of the BENTEE, then "a 
possible subdivision of VERBAL into a number of plausible bi'anches, then a 
possible further subidi vision of LITERATURE, then of POETIC ANALYSIS, then 
of METER, akd finally. to IAMBIC PENTAMETER, where we are obviously close to 
The content] of a specific test item. Thus" we observe that not many steps- 
Jre required (though the steps here may be altered with exploration) to 
move from the highest value-space of the judges (who represent "society'.') 
to the lowest and most detailed item-space or behavior-space. It is ap- 
parent, tod, that this tree is "recursive": that is, the same strategy may 
be applied at each sub-node, without any major philosophical shift, in mov- 
/Sig from such value-space to test-space. Furthermore, there is provision 
AfoMhe functioning of different sorts of "judges". At the top level, the 
y judges may be citizens, students, parents, board members, or? elected offi- 
^cials. At the lower levels, the judges may be (and indeed would almost have 
to be) psychological or subject-matter expert^ who thoroughly understood 
the -detail^ and t4ieiT-relative--importanee-wi-t-hin-the -t-opics-considered . - 
Thus the Ij^ntee provides for a reasonable movement from democratic princi- 
ples to technical expertise. 2 

* D. Two Techniques for Rendering Judgment 

There may be nuiuerous" ways of establishing proper weightings for the 
various nodes. y/One may be called the "token strategy" , jind the other the 
'icorrelation.^i strategy." In both strategies, one begins by ident\fying a 
correct s&i^ction of "judges", and by presenting to these judges a aescrip- 
tion of the sub-trakts which ma^e up the branches . from a particula.^ node of 
interest. For exanjple , in 'Figure 1, we might have brief descriptions of 
. the sub-fields SPEAKING, LITERATURE, READING, Gl'A^MAR, etc. Then we may 
ask our judges (actW independently of one another) .toy^weight these branches 
by one of these two tieh^i^ues : \^ 

Token strategy . Figure 2 suggests what the "token" strategy amsunts 
to: A judge is given a stack of tokens, cr poker chips, all of equal value. 



^It is interesting that use of the tree does not depend on "aemocratic prin- 
ciples" : the tpp-most allocation (or any subdivision) coul d be performea by 
any authorized person or agency. The bentee appears, ratner, a technical 
aevice for gaining control over the process of education and like most teen- 
nical procedures, is in itself quite value-free. 
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Each judga is given a staek of tokens, and asked to '^spidnd** tbtm aeeordiag to the 
importance or value of each topic. In this example, Judge J has awarded 2 to B, 4 to 

C» and 1 to 0< 




For each judge the ''value" of each topic is the number of tokens given, divided ^ 
the total N of 'tokens. 



si 



Figure 2. The direct-, or token, strategy for weighting, 
sub-areas in importance. 



Perhaps 100 such chips might be used. Then the judge is asked to "spend" 
the chips according to' his appraised values. of the subtraits. If 100 chip 
are so spent, then the number of chips for each branch inaicates the pro-- 
portion of importance given to that branch (within the node-family). Thus 
if 20 chips are placed on LITERATURE, this would indicate that .20 of the' 
value of VERBAL belongs (in the judge's opinion) to lit.erature study. And 
within each le 'al of the tree,**each node has branches which must sum to 
1,00 . The scale of such weightings is thus considered to be a ratip scale 
that is, 30 clfips would indicate "twice as much" importance as 15 chips. • 
Zero chips would indicate no importance at all for a particular branch. 

Cpn!eJLational strategy . As Figure 2 suggested the j udgmental situa- 
tion for the token strategy, so Figure 3 suggests the stiraula^^Bituation-— 
for theJborre.lational strategy* The instructions accompanying such bar 
graphs may be as follows: ' 

When we say a person is "well educated" or "well pre- 
pared," what do we mean" There are many wasy of making 
such a judgment, depending on what we regard as important. 
This present study is an effort to define what is meant 
by such terms. You^are asked to help decide, using as a 
guiae your own preferences and beliefs. /*^^ 

You are asked to judge 25 boys and 2,^girls, to rate 
them for how "well prepared" or "well educate^" they are. 
. All you will know about them is contained in a "profile," 
such as the one here. 

You are given this profile of the boy. You are asked 
to make your own overall evaluation of his preparation 
and education. These trait? are in a chance order (dif- 
ferent for every judge like yourself). It is up to you 
to decide which ones are more important for you. (Page, 
1972, pp. 41-42. For experimental instructions, see 
Appendix E. ) - ,9 
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Traits 



\ 



Profile of a Tw'elf th-giade Boy 




L QuiAtitative 


35 


******* 


*4^***** 


** 








2. Arts 


96 


******* 


******* 


******* 


******* 


******* 


** 




87 


******* 


******* 


******* 


******* 


** 






5 


******* 


**** 










' b}' Verbil 


90 


******* 


******* 


******* 


******* 


**** 




6. Personality 


41 


******* 


******* 


**** 








7. NAtural Sdencts 


20 


******* 


******* 


* 









1 



+■ 



Your Overall Jud^cnt 

^ I ^ I Ti ^ I ' I ' I 



10 



Very poorly 
prepared 



Average in 
preparation 



Extremely 
well'prepared 



Figure 3. " Profile of a Twelftji-Grade "Boy" which the 
Judge Evaluates by Checking af Spot along the Line of ^ ♦ 
"Overall Judgment" . '/ ' 

.^d these judges . like 'the token judges , are given descriptions of the 
traits', as a basis of making their evalu^ition. _ _ 



The principal difference between the strategies is that, in the cor- 
relational' method, the weighting of the traits. performed indirectly , 
through the individual weighting of individuaJ^ases. That is. the weight- 
ing may be inferred through discovering the col'relations between e^ch^it 
and the "overall judgments" made by a person or a ^oup. For-Wpresent 
the research generates these trait-profiles randomly by computer (the pro- 
gram written by Thomas Breen). and no inter-trait correlations are built 
into the profiles. J he 'resulting maf^rix' of intercorrelations would be ^ 
similar to that of Figure H, ^, . ' 

An interesting and unusual property oflU^ matrix is that the cor- 
relation coefficients in the last column are also the beta coefficients fo£ 
the l^ltiple regression R. for the best prediction of the "overall judgment" 
for a profile of traits. That is, 



r. 
ic 



^i 



(1) 



where i represents any trait, and c represents thie overall criterion or juag 
mnt , ana where Ci represents the ith beta weight for optimaUy predicting c 
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Traits 



Judged 
Overall 



I. Verbal 

-8. Quuitiutive 

3. Social Studin 

4. Hatutal SdeneM 

5. Arts 

6. Physical 

7. Persgntlity 



0 
0 



0 
0 
0 



0 
0 
0 
0 



0 X 

0 

0 

0 

0 



0 
0 
0 
0 
0 
0 



.30 

.25 

.22^ 

.20 

.15 

.10 

:io 



Figure 4. A Possible Matrix Generated by the Bentee 
.Judgments of One or More Persons, Using the Correla- 
tional Strategy. (The trait intercorreiations would 
be approximately zfero, reflecting the way they were 
generated. The correlations between traits ,and over- 
all ratings are hypothetical, for a possible individ- 
ual or group.) 
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(This is true only when the trait intercorreiations =0.) By the same 
reasoning, the multiple regression R for predicting c is 



R 



' i=i ^ " i=i 



r. 
ic 



(2) 



given n traits. 

Since these correlations are also beta weights, they may be assumed of 
ratio scale, so far as judged "importance" is concerned. That is, when one 
has a correlation of .30, this correlation, like a beta weight, may validly 
be considered as "twice as important" as a correlation of .15.^ In this 
sense, such weightings by correlation are similar to those of the token 
strategy. In order to make the two strategies still more comparable, a 
column of correlations may be normalized or "proportionalized" to sum to 
1.00. That is, each weighting may be redefined as 



^1 

\his equivalance will become intuitively clear when it is remembered that 
each beta coefficient represents the correlation of the trait with the cri- 
terion minus various products of the intercorreiations. Since the inter- 
correiations are assumed zero, all of these products are also zeroj and the 
correlation is left unqualified. 

'*Some may be tr^uljled by the fact that the amount of variance accounted for 
is not linearly represented by the correlation coefficient. Nevertheless, 
multiple- regress ion theory shows the beta weightings to be proportional to 
their magnitude of the coefficients. 
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i ■ . V •' 

' V/ = " (3) . 

.... j=l 

In this way, each V^^is the proportion of value for the ith trait ,,with4ji 
a given nocjie- family. ^ , 

i^* Te chnolog y of the Value T^ee 

Of course, to establish the -values of society, or of a profession, 
single judgments would not suffice; thus, one would average such proportions 
across a. sample of appropriate judges. And for each node in the value-tree, 
one woulcl establish proportions of value for the group which would also sum 
to 1.00 within each node-family. How would one thus appraise a student (or 
a group of students), given Vi and given a profile of his traits? At the , 
highest level, the appraisal of the jth student would be worth: 

n < , 
Bentee., = T(Z v.m. .) , » 
3 i=l ^ . 

where is the societal value of the ith trait , where m^j ,is^the measure 
(in standard scores) of the ith trait for the jth student, ana' where T rep- 
resents the transformation of the obtained sum of weighted measures to the 
T-distribution (the calculation of which would depend on the parameters of 
the norm-group to which the student belongs). And just the same techniques 
could be applied to each node in the value-tree. 

The appearance of this tree, where the values of the branches sum to 
1.00 within each node-family, suggests a fruitful analogy:, the probability 
tree ^for indepenaent events. Thus, in the value, tree, it is seen that the 
value of each node is the product of all the lineal values above it. That 
is, if LITERATURE is considered worth .20 of VERBAL, and VERBAL is worth 
.30 of the overall BENTEE, then LITERATURE is worth (.2 x .3 = .06), or 
Just 5% -of the overall 'BENTEE. ><ore generally, 

. L 

v^ = n V ^ - (5) 

k=l ^ 

w)iere is the overall value of the tth trait, where V represents tjie value 
of each direct parent-node between t and the BENTEE (or other selected par- 
ent-node below/the highest level) , and L is the number of such levels or 
generations b/tween t and the BENTEE. Thus a program is indicated for 
evaluating an^node at any depth in the tree. 

Overlapping branches . Yet reflection upon the tree of Fi^ure^ will 
lead us to another consideration: the overlap among the various branches 
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willt eventually, become- substantial.^ For example, if ARTS is defined as^ 
covering esthetic training, then we may expect that POETRY ANALYSIS may be 
reached through ARTS, as well as through ^V£RBAI». The weightings., of course, 
may be justifiably different within one value- family , as compared with an- - 
other. And whatjbver the value of behavior b turns out to be in one branch- 
ing, it may be added to the values of that same behavior b wherever it occurs 
in the vaiUe tree. Thus, , ' " 



H 
I 
hal 



'bh 



(6) 



where represeitts the overall value of behavior b (such as recognition of 
iambic pentameter) i 'where represents a specific hierarchy within the value 
tree) wherein behavior b is found i where H represents the total frequency of 
occurrences of behavior**bi and where v^j^ represents the judged value, of be'- 
havior b. in the branching h. Thus if the recognition of various, poetic 
meters Is worth .OOOZ in one setting, and .0001 in another, and those two 
occurrences are the only ones, its total value would be .OOOSt 

Dimensions of Concern # Still another procedural rule of thumb may be 
suggested by independent probai)ilities« Suppose we have two dimensions of 
concern, such as shown in Figure 5, suggesting four branches of social 



Taxonomy 
Levels 



Subject 'Matter Content Areas 



F«ct 


1 


.30X .10 
- .03 








Concept 


2 










Application 


5 






; 





.30 



.30 



.40 



.10 



.30 



.40 



.20 



Figure 5. Arbitrary Assignment of Cell Value in Tw.o- 
Dimensional Framework. = v^.v.j. (Where a cell's 

value is already estimated, that Estimate may still be 
used, and the others assigned by the product methoa. ) 



Anc the tree itself will of cours'b grow to great size. If we consider 
each node to have 7 branches, and guess that perhaps 8 generations will be 
adequate .to cover the evaluation tree, then we have 7^, or approximately 
million specific items or cells. But large numbers of these will be 
duplications^ reducing greatly the number of unique items. 

f 



30 



Studies and three levels of taxonomy » for example , fact, concept ^ applica- * 
tion* Thent we taay ^eigh each cell under the assumption of row<»and-c61umn 

. independence t by making the value of each cell the product of the row<*value 
times the column- value # (This strategy was suggested by Dr^ JosS Koig « 
IbffleSf GermSn Espinosa Htt and other Spanish educators at a seminar in 
Valencia in Junet 1972t) Where there is a judged interaction of the two 
dimensions t the already estimated cells nay be assigned, their estimated 
weights in advance, and the remaining cells assigned by a system of row- 

' column multiplication. . 

r. Management of tho Valuo Tree 

An earlier criticism was made of the linmanageable nature of the ^^botton- 
up" strategy, which begins near the ^^behavioral objectives" at the bottom of 
the tree. Is the top-down strategy of the^entee equally susceptible >t;o such 
criticism? Apparently not, for it offers procedures for diminishing. whole 
branches of behavior very early in the analysis , and reducing them to tri- 
vial size. Such pruning of the tree is -extremely important,' for it lets 
the educational system concentrate on those branches where substantial valuet: 
reside. * , 

Also, as more is learned about the intercorrelations of sub-goals, a 
large amount of pruning may take place in the measurable behaviors required. 
This suggests that the bentee weighting may function smoothly within a "ma-* 
trix-sampling" environment, wherein different students are measured with dif«- 
ferent items. 



For example, suppose an educational system is beginning to construct a 
new evaluation program ^om its beginning, and wishes to allocate^the efforts 
of scholars, psychologists^, and item writers to obtain the most valuable in- 
formation possible at minfmum cost. Then the b^entee seems to offer great 
utility, regardless of who the judges are , or how the/syste m is adrainistra-- 
tively organised . That is, if the educational system should happen to be a 
very hierarchical one, with highly centralized authority, then perhaps the 
Minister of Education (or some similar official) could, through his staff, 
allocate values at. the top levels, and draw upon experts or officials with-* 
in various disciplines to allocate values within their own lower subdiyis- 
ioi^s. In other words, though the bentee recursion pennits democratic ^samp- 
ling, it does not require it, and appears politically neutral* But it can 
balance claims of pressure groups of whatever persuasions. 

One difficulty of any top-down strategy is this: that the values of 
level 2, for example, cannot be precisely appraised until one has suffi- 
ciently explored levels 3, 4, etc. For example, the fact that PERSONALITY 
contains morality, at some lower level, may not be sufficiently realized by 
the judge at the top level. If he knew this, he might modify his evaluation 
of PERSONALITY at the top. Such a problem suggests that analogy of the 
artist painting a pictux^e: he usually allocates the major areas of his can- 
vas, and often brushes in an approximate color wash^ before he paints in the 
detail within the areas. In this way he -maintains artistic control of his 
canvas. Yet the subsequent detail forces the artist to make minor altera- 
tions in his various ^eas^ as h^ goes along. Similarly, the first BENTEE 
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evaluation may not remain the p6nn#nent allocatiotr bf top-level value, inodi 
ficat ions may be .made as the lower levels, are .further defined, ,andias their 
valuesj^eem to increase or diminish. Yet the 1:op-level allocation pemits 
the system to get underway. ^ 

Another objection sometiimes raised is that a top-down system may lead- 
to a rigid Procrustean bed of evaluation, which would override differing in 
dividual talents, and differing curricular or group goals. This objection 
seems unwarranted. Indeed, value trees may be created ^o^^^pecific needs 
of subgroups, sub-disciplines, or sUb-types within the system. A "bantee 
value tree for the diversity engineering student would expect ably differ 
fro^ that for the general high-school student. Even for the individual, it 
is conceivable that one sort of useful bentee, for purposes of self-direc 
tion and personal counseling, mighj be the sum.pf products of his trait 
measiirements times his personal allocations of value. .In this way ,j the 
traditional counseiing glS5-5f "self .actuali;^ation" might become a i^espect- 
able, technical possibility. ^ 

Properly understood, the bentee tree of values appears very close to 
what practicing educators have been doing through their present practices, 
which involve legislative rulings, curriculum conmittees, department un^r- 
standings, and teacher interpretations. What the bentee may do is to jTro- 
vide a technical rationaUzation^nd formal operating procedure for educa- 
tional values. Thus it might transform the current top-down system, which 
has "always seemed a natural "one to educators and to society, fran an intui- 
tive and uninformed strategy to one which is explicit, and quantifiable, 
and related in. clear ways to .behavioral objectives. Thus education may 
capitalize on the great efficiencies which such^ quantification of purpose 
has provided for other disciplines. 

Subsequent chapters will describe an investigation .into the practic- 
ality of discovering such social values for education, across differing 
social and professional groups. / 
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CHAPTER 3 
EXPERIMENTAL PROCEDURES 



' \ln Chap^r 2, the experiineKtai hypotheses were outlined. The present 
apt^XiJ^S^^ribes the SeKipling and' procedures used for the test of the ben- , 
e^j^thods. The goal was. to explore the values of society, especially as / 
reflect^4 in two groups i educators and non-educators. Thus» it was thought, 
desii'abl^ to stratify the sampling of the judges. In order to measure their 
reactions\ it was neiCessary to write coinput.'sr programs producing the stimuli 
and analyzing the data. This chapter^wiil explain the problem$ of sampling 
encountered, the nature of the experimental treatments, the coinputer pro- l, 
grams for producing the stirtiuli ^ ai;d the .techniques of data preparation. ^ 

Sampling . The original sample design called for 100 subjects of whom _ 
50 were to be professional educators engaged in secondary cduc'ation and 50 
lay people, who were' not profess ipnally' connected "with secondarj/ education. 
The actual number of subjects recruited wasylOl of whom 52 came frcm the 
professional and 49 from the lay group^. Tab^e 3 contains a li'st of the 
areas in which sample teachers worked."* \ \ o 

Half of the ssunple (51 subjects) was selected from Connecticut, mainly 
«^from eastern Connectiput, from areas convenient to the University; while 
the other half (50 subjects) was from the greater CleveJaiid, Ohio area, 
^here another researcher assisted with the data collection. Research of 
this kind often ^inds no significant ^ffect due to region" of the country. 

Major effects are due, rather, to social -cia^s^ type variables.-.- Sampiin&^ _ 

from two regions edlowed us, how even', to test this assumption,, although it 
may be confoj^ded with pny possibl^effect due to researchers. 

The sampling process was designed to take about a half-hour of^ each 
subject*s time. It took from 20\to 40 minutes . " From the lay grou^it took 
place in their homes or places of^empioymcnt . For the profess iOna%group 
. it took place in the schools where they worked, althougi several carai^ to 
-the researchers' offices for the session. Information was collected about 
the subjects' age, sex, race, education, and location (Cleveland or Con- 
necticut). Table 1 contains a breakdown of the sample by Occupation, sex, 
and location. All subjects were x-eimbursed for their time from funds 
awarded for that purpose by the^ Undated States Office of Education. 

The high school teachers. and administrators were confacted through the 
schools v;nere theyo worked. Although schools, were selected ranaoniy, the 
selection of teachers was most often made by the principal's office and thus 
cannot be considered a random selection of teachers within the school. They 
are probably friends of the principal or teachers who answered a rec^uest for 
volunteers made at a staff meeting. The specialization of the teachei-s in 
the sample are listed in Table 2. The three teacher-educators were selected 
from among doctoral candidates completing their degrees at The University of 
Connecticut. . ' 



An attempt was made to recruit members of the^iay sample by calling 
randomly selected telephone numbers. A standard fonnat was designed for 
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' • f TABLE 1 
OCCUPATION, SEX, AND LOCATION OF SUBJECTS 











Cleve. 


Conn. ■•• 


Professional 


.Teachers 

Administrators ^ 


Female 


' ■ t / 


13 

8 

5 


15 ■■. 

5' 

3 


.-J 


Teacher-Educators 








.••3 


Lay 


Women 






11 


10 




Blue-Collar Men 






6. 


7 




White-Collar Hen 






7 


8. 



TABLi: 2 

SPZCIALIZATldNS OF SAMPLED »TLACHEPsG 



Social Studi^BS 


9 


Art 


1 


Lnglish 


"6 


Bookkeeping 


1 


Mathematics 




English 6 History 


1 


Biology 


u ^ 


Clothing & Tailoring 


1 


^^usiness L ducat i<m' 




English 6 Spanish 


1 


History 


3 


French S Spanish 


1 


Physical Education 


1 


French 


1 


Science 


1 


Pr "dcs 


1 



this telephone conversation which, it was noped^ would insure the greatest 
amount of cooperation. However, this method proved to be a failure* Ap- 
pendix A contains a copy of the ill-fated format for the telephone conver- 
sation^* 

The refusal rafe for the first lU calls was over 90 per cent (13 out 
of 1^). There was a number of reasons for this* The first contact spoke 
no English, A number simply said they were not interested. One woman 
could not discuss participating in the studj^ unless her husbana was pres- 
ent. One prospective subject wanted to know exactly why he had been chosen 
as a subject. He did not believe the professed intent of the research ana 
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seemed to feel the researcher might be an agent of some investigative agency* 
At a couple of numbers there was no answer* When the. r^ijfie.arphen did visit ' ' 
the home of the one person who had agreed over the telephbn'^ to ^ake part 
in the study, h^^ found that the man had recruited his jcoXleg^-educated son ' 
to stand in for him because the son could do a **better job*" The researcher 
in Cleveland had s|.milar experiences* 

Thus, the telephone approach'H^as dropped in f avor ,of one ..that could re- 
cruit more people* , Contacts were made with some individuals through profes- 
sional, social, and religious affiliations, or through their place'^of employ* 
ment* People who were willing were recruited for the sample and were visited 

by the researcher at their homes, or at times, where they worked* Contacts. - 

were also made through some of these subjects with their jfriands, neighbors, 
and co-workers* In this way the sample was completed* 1 

Since the sample was not selected randomly, it may be possible that the 
sampling process has in ^ome way influenced the results of the study* If ^ 
the study had been an attempt to construct the final measurable ^objective 
for education, the exact social values of the traits would have been of 
prime importance and the lack of a random sample most- certainly d^aging* . 
But since the purpose of the study was more to test and develop a methodol- 
ogy, an appropriate sample was one that included an "adequate number of people 
from professional and lay populations to extract reliable weightings of the 
traits* With 'these it would be possible to begin investigating differences 
in their educational values* This type of investigation is not invalidated 
b^' the selection of a non-represencative sample, although the generaliza-> 
bility of the value measures may well be affected* 

Profiles * Eadi of the subjects was asked to evaluate 50 , hypothetical ^ 
tvelfth-grade students, 25 "boys" and 25 "girls," according to how "well- 
educated" or how "well-prepared" each one appeared to be* The definitions 
of the terms "well-educated" and "well-prepared," which were used synony- 
mously, were not specified* Subjects were encouraged to rate students ac- 
coii'ding to their own values* These hypothetical students were presented 
to subjects on profiles produced by a computed program* The format of the 
standard^ prof ile appears in Appendix B* 

"Ko attempt was made to produce realistic profiles* Although some tech- 
niques for generating Tmaltivariate data with desired realistic interrela- 
tionships have been suggested (Capra and Elster, 1971), it was considered 
more informative to generate student profiles with no correlations between 
the traits* If there were systematic relationships between the profile 
traits, it would be extremely difficult to estimate the uiiique importance 
of each trait in the formation of the subject *s overall judgments* 

Some evaiuators, generally from among the more sophisticated profes- 
sionals, did remark about the unexpected lack of relationship betweeiv the 
traits 8 They were told that for pm^poses of the study no attempt was made 
to produce realistic profiles* 

In order that extraneous details of the profiles not systematically 
affect subjects' ratings j> the trait titles and scores were randmly 
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ordered for each subject's set of profiles. Also, the order of the sexes 
within the profile sets was randomized «- 

Program PRQFrLE * A computer pro-am, called PROFILE, was developed to 
generate these randan student profiles. It was written for the IBM '360/65. 
in The University of Connecticut Computer Center in the IBM computer lan- 
guage PL/I, level F <-IBH, 1969, and Weinberg, 1966). It is a progron which 
generates large numbers of profiles of the form required for this study. 
The program ^allows the user to specify up to ten traits for each profile. 
It also produces punched card output, one card for each profile cont.''iing 

"the profile identification data, trait order on the profile, and trai t 

s-cores.- ' . _ ' . ■ - --• 

The PROFlfi: Program was written because it was the only feasible way 
to mass-produce five thousand unique student profiles, half boys and half 
girls, with a random trait-order^ for each set, and with randomly generated 
trait scores. At the same time it could automatically produce, w^th great 
savings of time and manpower, the punched cards needed for later analysis. 

?>■■■ ' , ■• 

PL/I was chosen as the language' for the program' because of the ease 
with which it allows for the manipulation of strings of characters. This 
was very important since .the profile, format contains a diagram i normal 
curve, nultiple printing oh the ahae line, and variable length bar-graphs 
for the T-scores. These specifications were easier to program in a list- 
producing language like PL/I than in a mathematical formula language such 
as FOi^TRAN. . „ • 

Because of the input^output restrictions of the University computer i 
the program had to be designed to produce the profiles in small batches « 
Input and output devices, Specially the 'printer , are extremely slow com- 
pared with the b^eratioh of the central processing unit. The operation of 
the system would be greatly impaired if the input/output devices were taxed 
to capacity. Output is stored ten^>orarily on a magnetic disk and is printed 
or punched when the program is completed. The storage capacity of the mag- 
netic disk is far less than what would be needed for the 5|000 (100 evalu- 
ators X 50) profiles that were generated, so only a specified number was 
produced on each run. 

since each profile and its data card have a unique identification nxim- 
ber, the ^, first number in the current sequence has to be included as input 
to the program. Also included in the input parameters a^re the number of 
sets of profiles to be generated, the number of profiles for each set, the 
number of traits to appear on the profiles, and a list of the trait titles^ 

The program is not specific to the requirements of the present study • 
It was designed to be readily usable by other researchers, one of whom is 
cu rently using similai^ profiles for a study of his own. 

Appendix C contains a flow chart of the program • And a listing of 
Program PROFILE is available. What follows is a detailed description of 
the program. ^ 
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Program Flow. The program first allocates suf<i/bient menory ^pace for 
storing all necessary arrays and the profile printout format; then it reads 
the input parameters and the trait titles. After these preliminaries it 
initiates the r^dom- number sequence by reading the computer clock and con- 
verting^ the time, which ^appears in hours, minuted, s'econds, and milliseconds 
to an odd number. This large number becomes the seed number for the sub - 
rbutine which produces random^*numbers for the^rest of the program. 

Next, a counter begins which will check the number of sets of profiles 
produced by the program gainst the number requestea on the parameter card. 
This counter serves as the evaluator juig^er. It begins with the evaluator 
identification number ipeeified in the input register and increments until 
the correct number of profile sets has been generated. 

For each of these sets two batches of profiles will be produced, one 
to be labeled "twelfth- grade boy" and the other "twelfth-grade girl." The 
number in each ba:tch will be the same and is specified by the user on the 
Input cards. The order of these two halves is random. It is decided by 
selecting a random number B^ween zero and one. If thi.s-^umber is greater 
^than one-half (.5), the first half of the profiles is called boys; if it 
is less than one-half, the first half is girls. 
» ■ ' 

Also, the order in which the trait titles are printed is determined 
randomly before a profile set is produced. Random numbers are computed be- 
tween one and the number of traits. After each number is computed it is 
assigned to the next trait in the original trait input order. As numbers 
are assigned to traits, the program checks back through the traits for any 
duplications. If a sequence number duplicates a previous number, the trait 
being sequenced receives the next random number. This process is terminated 
when each trait has a unique sequence number. 

At this point a second index begins to count the number of profiles in 
each half-set, until the specified number should be reached. Then the cal- 
culation of the percentile scores for the traits begins. The program gen- 
erates as many random percentiles, between one and 99, as there are traits. 
Each percentile is generated from a random number between zero and one. * 
The corresponding T-score for the profiles is calculated by a set i^butine 
from the same random number, since the T-score is a function of the area 
under the normal curve, of which the random number can be considered a 
measure. The percentiles and corresponding T-scores are assigned to the 
traits in order of original input sequence. 

The completed profile is then printed out. A routine prints the stan- 
dard format when all information is complete. The appropriate sex is 
pointed at the head of the page, ana the trait titles, percentiles, and 
barographs for the T-scores are printed in the randomi.-ed order derived at 
the beginning of the sequence., An identification number, composed of the 
evaluator number, the sex identification number, and the profile sequence 
number, is printed at the bottom of the page. (Reference may be made to 
Appendix B for an example of the completed profiles.) 



Next a data card is punched. This card contains a one-digit code, 
which can be used to identify the set of traits used on the profLle, the 



same composite identification number as the profile, the trait order (the 
sequence number of e^ch trait on the profile), the percentiles, and the T~ 
scores. This information is produced for the traits in the original trait 
input order, not the order in which it appeals on the printed profile. 

After punching the card, the program checks the profile index to see 
if all profiles for the current sex group have been produced* If they have 
not I the index is incremented and another profile producible When the num- 
ber is complete I the prograsn checks to see if profiles have been generated 
for both sexes. If nott it will change the se^ -and begin another set after 
resetting the profile index* If both sexes have been printed » it will check 
to see if it has produced the number of profile sets requested by the user* 
If this check is false^ it increments the set counteri i^e*, the evaluator 
number t and begins the operation again by calculating new first sex and new 
trait sequence. 

When all the profiles requested have been produced ^ the program will 
Attempt to read a new set of input parameters* More parameters and trait 
titles can be read into the ccmpuer to' produce profiles with a different 
set of evaluator r-jxnbers, or proxies with different traits* A new input 
set will cause the progrsn to begin at the initiation of the set counter* 
Othezvise, the program terminates after printing a message that no more in- 
put cards were encountered* This infoms the user why the prograsn has ter- 
minated* 

Program' PROFILE was used to ,print all the profiles needed for this 
project* However, siwiceNlt has been writtctn as a general program , it can 
be used by anyone attempti^ng to do similar kinds of researcht or by anyone 
wishing to extend the scpp^ of the current project* 

The Traits * By nature »\the information-processing capacity of the 
human subject is limited* The^^^verage perspn may be able to process simul- 
taneously about seven pieces of information without undue confusion (Miller , 
1956)* For this reason, the number of traits was limited to seven* Seven 
headings were chosen which could include all aspects of the curriculum so 
that the amount of Information could be readily manageable* 

The seven traits chosen are arbitrary* They are designed to include 
in a logical fasion all aspects of the curriculum* They are a redesigned 
version of the seven proposed for investigation by Page (1972) and thus 
represent only one possible logical analysis of the edHcational program* 
7lm logical breakdown had been proposed to various educators and researchers 
prior to its adoption in the study, and no reservations or criticisms were 
expressed* 

The trait titles selected are the following: Quantitative, numbers 
and related studies; Arts, esthetic and practical studies; Physical, health 
and hygiene; Social Studies, civics and social sciences; Verbal, communica- 
tion skills and verbal reasoning; Personality, social hehavicv and character 
development; and Natural Sciences, the physical and biological sciences* 
The complete trait descriptions which were given to the evaiuators appeal^ 
in Appendix D« 
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As yet', there has been no comprehensive attempt to develop the com- 
plete structure of th€ bentee under these headings • But whent and if, com- 
pleted,^ this structural would extend from the value-space down to particular 
operations and behavipral criteria* The importance of eacii operation and 
criterion in terms of the ove'r-arching goals and ^yBlue^pf education would 
be computable from the product of the weight^ of each operation t sub-trait, 
and trait of which it is a part. These characteristics,/ wore noted in Chap- 
ter 2. ; \ 



One reservation must be made regarding the traits. Although there wore 
no complaints expressed during the research about their theoretical complete- 
ness, either by professional researchers or by the subjects, it can be argued 
that another category should have been included: Vocational education. It 
was felt that vocational courses could be included under 'one or more of. the 
other traits without * overly stretching their definitions. That the absence 
of an additional trait went unnoticed may indicate general acceptance of the . 
logical scheme. ^ 

>• ~ „ ' • 

^ P^ta Collection . The data were collected in a personal interview with 

each subject. Each session lasted approximately 30 minutes (20 to min- 
utes). Tl^e length of the session depended on how quickly the subject under- 
stood the instructions and how quickly he was able to work through the mate- 
. rials. 

At the beginning of the session, subjects were asked to fill out ident- 
ification sheets which :asked for their name, address, phone number, age, 
sex, education, and occupation. Each information form 'also contained space 
for the subject's identification number, which was filled in by the re- 
searcher. The researcher also made note of the subject's ethnic grouping. 

« 

After the form had been completed, the researcher explained the gen- 
eral purpose of the study and gave a brief explanation of the material, the 
profiles. The standard format for the interview instructions appears in 
Appendix £• These instructions could be modified by the researcher at his 
discretion to facilitate the subject's mastery of the task. 

In conj\Snction with the description and explanation of the profiles, 
the subject was given, on index cards , brief descriptions of each trait. 
Since the arrangement of the trait titles on the profiles was random for ^ ^ 
each subject, to protect against the possibility that subject?' evaluations 
could be influenced by trait order, the descriptions were printed on cards 
so they could be rearranged at each interview session into the same order' 
as the tratis appeared on the computer printout • 

After the subject felt he was sufficiently familiar with the trait de- 
scriptions, he might have been asked to assign weights to the tratis by 
sorting 50 pennies according to how important he felt each trait to be. 
The subject was asked to use all the pennies , but was free to place them 
where he wished. 

Subjects were asked to make independent assignments for each sex. The 
order of the sexes for this evaluation was the same as the rai^dom order in 
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which the profile packet had been arranged by the computer. Half the sub- 
jects, by random selection, performed this task before the -profiles were _ 
explained to them. The other half perfonned it after they had completed . 
their profile evaluations w 

After a subject had read the trait descriptions and assigned the pen- 
nies, if that were called for, the researcher explained to him the profile 
format. He explained what percentiles and T-scores were, pointed out that 
half ^ the profiles were "boys" and half "girls," indicated the scale under 
the profile for recording his overall judgment, answered any questions, and 
offered any further explanations until the subject understood the task. 

Subjects were then taken through a sample profile and shown how to re- 
cord their judgment ,' what kind of mark to make-, and where to put it on the 
computer printout. After the subject .knew what to do anc understood the 
profile^ and trait descriptions, he was given the profile packet and left, 
undisturbed to maJ'.e hi^.. evaluations. The trait descriptions were, left with 
him for any necessary reference. 

When the subject had completed the 50 profiles, the researcher col- 
lected them. If the subject had not weighted the traits prior to evaluat- 
ing the profiles, he made his penny assignments at this point, and his 
wei^tings were rrrorded. 

The researcher then paid the subject his stipend and, if the subject 
were interested, discussed the study with them. Any significant comments 
or suggestions were recorded. 

Data Preparation . First, the profiles wex-e scored by hand. Each eval- 
uation iriirk made along the judgment continuum under the profile was scored 
as a number from 1 to 81. There was no intrinsic or generalizab.le meaning 
to this numbering system. It was arbitrarily ddopted because the judgment 
continuum line was composed of 81 segments. Scoring by line segment number 
was the least taxing and least fallible method which allowed making- more 
than a ten-point discrimination in rating. The rating scores were then 
punched onto the data cards which had been produced when the profiles were 
generated. 

The information from the identification form was also coded and punched 
onto another data card. This card contained subject number, a code for 
location (Connecticut or Cleveland), age in years, sex, race^ nunfcer of 
years of education, stratum number, and codes indicating whether the sutject 
performed the token assignment before or after evaluating the profiles and 
whether his profile set contained "boys" or girls" first. • - 

This data card also contained the token -weigh tings assigned to each 
trait by the valuator. This card and tb' set of profile .cards contained all 
the data used in the analysis. From the^e data all Variables used in the 
analysis were constructed. The next chapter describes the computation of 
those variables and the results of the statistical analyses performed. 
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• CHAPTER 4 . 
EXPERIMEMTAL RESULT'S USING .THE BENTEE 



The previous chapters have suoimarljsed the theory concerning the bentee 
and have outlined the procedures employed, and the sample of judges drawn 
for the experimental work. Still, it has not been established that human 
judges may reasonably serve such a function. Is there any reliable consen- 
sus of people? Do different groups of judges differ much in their evalua- 
tions? Do bentee evaA^uations vary according to student sex? These and 
similar considerations occupy the present chapter. ' \ 

The contents of this chapter will be as follows: A) Results ^^f the ' 
token method. B) Results of the profile or coxrelation method. C) Com- 
parisons of the token and profile methods • D) Values related to- personal 
variables of the ^subject- judges. £) Factor analysis of the value matrix. 
Apd jT summary will c^onclude the chapter* 

^ A. Results of the Token Method 

/ As already noted, each judge was asked to evaluate the imp^.^tance of 
seven tratis for the education of boys, and the same seven tratis for the 
education of girls* Then for each judge, an average may be calculated, 
combining the weights given for the two sexes, and this is the judge *s 
total evaluation. Thus there are 21 token*assignment variables for the set 
of judges, and a summary of these spears in Table 3. 

The order in which traits were presented to subjects was random; in 
the analysis they are presented in all tables in their order of importance, 
as computed by the combined weights received in the token assignment method 

Each set of seven variables forms an ipsative measure. An ipsative 
measure is one in which the sum of all the values is equal to a constant. 
Since subjects were given 50 pennies to assign differentially to the seven 
traits, the sum ^ of all the token assignments for each subject and the sum 
of means across ^groups of subjects must always equal 50. It is obvious, 
then, that the number of tokens assigned to a trait cannot be considered 
a measure of its absolute value. It is thus impossible to make comparisons 
between subjects; i.e., it is impossible to say whether Verbal is more im- 
portant to Subject A than to Subject 3. The meaning of the n\iraber of 
tokens assigned to a trait is -relative to the other numbers within the 
same ipsatiye set (Clemens, 1965). 

As was expected, the means for these judgments about the relative 
values of the traits are not equal* Also, the values of the traits for 
both sexes are not the same. The judged order of the traits in importance' 
is not the same for the two sexes. And even where the traits appear in 
relatively the sane order, there are differences ii\ the relative weights 
of the traits. 



TABLE 3 . 
TRAIT WEIGHTINGS BY TOKEN METHOp 



Ti-ait 



Mean 



S.D. 



Boys 



Girls 



Maximum 



Minimum 



Verbal 


10.21 


4.87 


44 


0 


Personality 


. .8.19 


4.37 


22 


0 


Quantitative 


7.71 


2.71 


15 


1 


Social. Studies 


. 7,10. * 


3.12 


20 


0 


Natural Science 


6.28 


2.33 


11 


0 


Physical 


5.98 


3.22 


19 


0 


Arts ' 


4.41 


2.44 


10 


, 0 



Verbal 


10.44 


4.91 


44 


I 


Personality 


8.94 


4.54 


25 


0 


Quantitative 


6.56 


2.74 


• 13 '■ 


0, 


Social Studies 


6.89 


2.88 


20 


0 


Natural -Science 


5.57 


2.45 


11 


0 


Physical 


5.52 


3.23 


19 


0 


Arts 


5.98 


2.62 


15 


0 


< 

smbined 










Verbal 


10.33 


4.83 


44 


2 


Personality 


8.b6 


4.22 


20 


0 


Quantitative 


7.14 


2.51 


13 


1 


Social Studies 


7,00 


2.78 


20 


0 


Natural Science 


5.93 


2.26 


11 


0 


Physical 


5.75 


3.12 


19 


0 


Arts 


5.19 


2.22 


10 


0 



Means are computed over the 101 subjects • 



There seem to be some slight biases in subjects' judgments about the 
relative importance of the various traits for boys and girls • Although 
Verbal and Personality were considered most important for both sexes « they 
appear to be judged relatively more important i compared to the other traits^ 
for girlse Quantitative was judged third in importance for boys, but Social 
Studies was considered more important than it was for girls • And Arts was 
judged least important for boys^ but more important than either Natural 
Science or Physical for girls ^ 
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As can be seen from Table 3, there were great differences of opinion 
anvong subjects about all the traits\ indicated by the large standard devia- 
tions and the extreme ranges of maxiniuin and minitnum values. For all the sex- 
trait combinations^ except Quantit<ative for boys and Personality for girls ^ 
there was at least one subject who rated the trait of no^ or of absolutely 
miuimaly importance. \ 

For any one trait the differences caxiXbe quite drastic » witk the trait 
being judged of least importance by one siib ject ancl of greatest importance 
by another The, maximum values do fall^ hcweviervr.into approximately the 
same order as traits were judged in importance^'b^^|^ie sample. This rela- 
tionship and a "timilor one for the standard *dc^X|dtl^cnd may indicate that 
the bulk of the mean^ differences ir^ judged traitv values may be due in larg- 
est part to some subjects who had strong opinions^ about the trails. 

A good number of the subjects i:ended to rate the traits of almost equal 
importance, making only slight distinctions between them. Many of them gave 
all the traits just seven tokens except for one judged slightly more insert- 
ant to which they gave eight. Such subjects many times would simply change 
the location of the' one extra token when they made their judgment is for the 
opposite sex. Many gave indications that they would have been more satis- 
fied had it i.^een possible to weight all traits equally. The differences in 
means for the traits is certainly more a reflection of the judgments of 
other subjects with stronger opinions, than of these meek subjects. ^ 

Such differences in the opinion and rating behavior of the subjects 
may have been related to the group to which a subject '|>elonged. Subjects 
who were professionally engaged in some aspect of secondary education may 
have valued the traits differently from those who were not engaged in edu- 
cdtion# Sp the sample was broken down to the two strata, and mean weight- 
ings were calculated for each trait. Teible 4 contains this breakdown. 



TABLE U 

TRAIT TOKEN WEIGHTS 
BY TRAIT, SEX, AND STRATUM 



• Professionals Non-Professionals 





(N = 


52) 


(N = 


49) 


Trait 




Girl 


Bo^ 


Girl 


Verbal 


9.94 


10.19 


10.51 


10.69 


Personality 


8.08 


8.77 


8.31 


9.12 


Quantitative 


7.63 


6.35 


7.80 


6.80 


Social Studies 


7,15 


6.98 


7.0i+ 


6.80 


Natural Sciences 


6.37 


5.77^ 


6.18 


5.37 


Physical 




5.5] ^ 


5.92 


5.53 


Arts 


U,77 


6.UH 


4.02 


5.49 



43 

Differences between strata do not appear very large , Both groups of 
subjects seem to exhibit the possible aex bias; mentioned above. The non- 
professional subjects seem to rate Verbal hifaer than do the professionals 
and Arts correspondingly lower. Other diff»9 ranees appear negligible. 

An analysis of variance was perfc^med to test whether these observed 
differences between the trait weightings were statistically significant, 
A model \t'as chosen which would test the possible effect of sex of hypotheti- 
cal student oh judged trait value and also teatt- for differences in rating 
behavior between professional and Liy subjects, , 

The design chosen for this analysis o.f vari=ance^ is a three-factor re- 
peated measures design adapted from Winer (.1S32), The three factors are 
a) stratum, with two levels, professionals and non-professionals; b) sex 
.of the hypothetical students, also with t»/o levels, boys and girls i and c) 
trait, the repeated measure, with s.even levels. 

Some reservations must be made about using analysis of variance with 
the token assignments. As has boen mentioned, the numbers of tokens as- • 
signed to each trait are not independent. They constitute an ipsative mea- 
sure, since. the sum of the seven traits weighting is equal f. 50. No prece- 
dent has been found for this type of analysis, and Clemans (1965) makes no 
mention about analyses of variance with ipsative measures, 

A repeated measures design was used fwpHTfiie analysis since it was one 
way to attempt to control for the^ct^^^t the ipsative measures were not 
independent of each other. The repeated measures design applies in situa- 
tions where the saii\e subject responds to a variety of stimuli, here the 
trait descriptions, and his scores may not be independent. It seemed to be 
the Resign best suited to test the^differences >bserved in the token weight- 
ings. 

Since the data were ipsative, finding any significant effects for 
either of the first two factors, stratum or sex, would be impossible. Since 
each subject was given 50 tokens to assign to the traits for eacn sex, his 
total score would always be 100 tokens. The score for each stratum would- 
be 100 times the number of subjects , and the score for each sex would be , 
50 times the total number of subjects. 

The. meaningful tests are for differences between trait value for the 
interaction of hypothetical\ student sex and trait value, and for the inter- 
action of stratum and trait lvalue. In other words, the analysis tested 
whether there were significant differences in the weightings derived for 
the traits, whether the relative differences in tjfre weightings were similar 
for boys and giria , and whether the weightings of ^the traits differed for 
professional and non-professional subjects. 

The summary t'&ble for the analysis of variancfe of the token weights 
appears in Table 5. As has been mentioned^ it is ^ed on the model which 
appears in Winer (1962, p. 328). Since there are ul^qual numbers of sub- 
ject in the two strata, appropriate changes were mades^" some denoninators 
and aegrees of freedom in Winer's model. 
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TABLE S " 
ANALYSIS OF VARIANCE— TOKEN tfEIGHTS 



Source 
Between Subjects 
Stratum 

Subject within group 
Hi-rhin Subject 
Sex 

Stratum x Sex 
Sex X Subject within 
group 

Trait 

Stratum x Trait 
Trait x Subject within 
grotip 

Sex X Trait 
Stratum x Sex x Trait 
Sex X Trait x Subject 
within group 



Sum of Squares df 
32.7 100 



.3^ 
32.36 

20,687.2 

0.0 
. 0.0 
2. If 



3,917.5 
65.07 
15,06^.13 



260. ^ 
2.84 
1,374.86 



1 
99 

1313 

1 

1 

99 



6 
6 

594 



6 
6 

594 



Mean Square 



.34 
.33 



0.0 
0.0 
.02 



652.92 
10.84 
25.36 



43.40 
.47 
2.31 



1.04 



0.0 
0.0 



25.75'W"'' 
.43 



18.75»'"'"> 
.20 



ititit 



p < ..001 



The null hypothesis of no differences between trait values was re- 
jected at well beyond the' .001 level of significance. There definitely 
were differences' in the relative weightings assigned to these seven traits 
by the direct token* method. The null hypothe^a^s^r the sex-by-trait in- 
^1:eraction was also rejected at well beyond the .001 level of significance. 
So sibjects felt that some traits wer^e more important for students of one 
sex than for those of the other. 

There were no other significant effects or interactions. Stratum had 
no effect at all. There seem to be no differences between the sample of 
professional educators and of lay perople in the way they respond to di- 
rect evaluations of the traits. 

The variance between subject and between sex was detennined to be zero, 
due to the ipsative nature of the measure. That some variance appears in 
these two factors is due to a slip in arithmetic. For a couple of subjects 
the numbers of tokens recorded for the seven traits did not sum to 50. 
These subjects. ei|neK were given more than 50 tokens or their responses 
wer^ recorded incderr^tly. 
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The results indicate' that by a direct assessment method, in which 
judges must assign tokens in an explicit way, statistically significant ) 
weightings can be obtained for a set of academic traits. Subjects' trait i 
values,. do not seem to be a function of their professional relationship to 
educ^ation. The profile of trait values for professional subject did not 
differ significantly from that for non-professionals. 

'"■ ■ ' 

There was definitely a sex bias operating in the assignment of the 
tokens. Personality and Arts were rated more important for girls, and Quan- 
titative and Natural Science ratec less so. The bias may represent what 
different training a, girl shovild have, to be able to overcome the -stereo- 
types. It does show that when, aske'd directly, subjects do not see students 
as the women's liberation advocates would want, i.e., with no distinctions 
on sex* • ^ « ^ 

B. Results of the Profile Evaluation 

Profile Evaluation Method . The trait values have so far been presented 
in terms of a direct assessment of trait by assigning tokens. In such a di- 
rect fashion, subjects might well be influenced by expectancy of what others, 
including the researcher, might think. Or they might respond to what they 
believed their values Would be, although in practice, in making judgments 
about students, those values might turn out to be quite otherwise. Thus a 
second method was used to maasure the values of the same group of subjects 
in <i somewhat mpre lifelike situation. This was done by having subjects 
evaluate a group of student profiles. 

Chafer 3 described the computer generation of random student profiles. 
These contained random percentile scores with their T-score equivalents in 
a random trait seq^uence to avoid any possible sequence effect. Subjects 
evaluated 50 profiles, 25 "boys" and 25 "girls." The scores appearing 
on each profile were compared with subjects' overall judgments by computing 
Pearson-product-moment correlation coefficients. Forty-two correlations ' 
were calculated. 

There ,are_j:hree sets of these correlation coefficients. Each set con- 
tains 1^ coirelatio^s , seven computed using the trait percentile scores and 
seven using the trait'^i^i cores . The first set contains the correlations 
calculated on the set of "boys" profiles, the second the correlations fran 
the "girls" profiles, and the third the correlations calculated for all the 
profiles. The four sets are contained in Tables 7, 8, and 9. 

As was stated in the section on procedures, Subjects were given 52 
profiles to evaluate, 26 "boys," and 26 "girls." However, not all the cor- 
relations are computed. over 26 or 52 profiles. The correlations are for 
all profiles for which valid judgments had been recorded. This number is 
usually 26 or 52, but for seme subjects there were fewer valid judgments, 

Th^re are a number of reasons for these omissions. First, on some 
occasions during the explanation of the task, so much use was made of an 
example profile that it was discarded lest its evaluation have been biased 
by the d'tscussion. Second, there were some profiles which were garbled 
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TRAIT WEIGHTINGS BY CORKTLATION METHOD 
"BOY" PROFIl£S 



Trait 


Mean 


S.D. 


Maximum 


Minimum 


Verbal T 
^ Verbal % 


" > .396 
,400 


•."^^35 
.234- 


.921 
.933 


-.342, 
-.330, 


Person T ' 
Person % 


. .319. 
.324 


.282 . 
.284 


.935 
.948 


-.365 , 
.-,359 


Quant T 
Quant % 


- .286 
.290 


.254 
.259 


.600 
.825 


-,375 
-,348 


SocStu T 
SocStu % 


.232 
.236 


' ''^ .251 
.250 


.788 
.790 

.607 
.616 


. -,267 
-,274 


NatSci T 
NatSci % 


.197,-. 
.204 " 


.219 
.221 


*^-,382 , 
-,387 


Phys T 
Phys % 


.125 
.126 


..255 
. .258 

i 


.741' V 
.729 


-,459 
-,450 


Arts T 


.137 
.136 


. 207 

.213 ^ 


.711 
.708 


. -,386 
-,418 


Each mean 


is the arithmetic mean for 101 subjects. 
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TABLE 8 

TRAIT WeI^TINGS BY CORRELATION 
"GIRL" PROFILES 


METHOD' 




Trait 


Mean 


S.D. 


Maximum 


Minimum* 


Verbal T 
Verbal % 


.435 
.439 


.218 
.218 


.908 
.929 


-,120 
-,101 


Person T 
' Person 


.325 
.327 


.270 
.268 . 

• 


.925 
.947 


-,237 
-,227 


/ • 
Quant T 

Quant % 


.260 
.26? 


,217 
.219 


.785 
.791 


-,274 
-.208 


SocStu T 
SocStu % 


.229 
.231 


,241 
.242 


.737 
.758- < 


-.408 . 
. , -.414 


NatSci T 
NatSci % 


.195 

»i95 


•o220 
,222 


.724 
.723 


-.369 
-.406 


Phys T 
Phys % 


.142 
.147 


.240 
,240 


.735 
.725 


-.411 
-.417 


Arts T 
Arts % 


.173 
.169 


,244 
.246 


.724 
.732 


-.299 
-.263 




.. , TABLE 9 

TRAIT WEIGHTINGS BY CpRRELATIOM METHOD^ 
. -AhL PROFILES COMBINED 

1' 



Trait 


Mean 


S.D.> 


Maximum 


Mininuin 


Verbal T 




.189 


AAA 

.922 


-•llH 


Verbal % 




.190 


.937 ^ 


-•086 


Person T 


.32*1 ' 


.235 


.924 


-.175 


Poroon % 


.325 


*234 


.945 


-.176 


Quant T 


.274 


.198 


.729 


loo 

-.183 


Ouant % 


.27U 


. 2.02 


.730 


-.203 


SocStu T 


.230 


.196 


1707 


-.137 


SocStu % 


• .231 


.197 


• .701 


-.157 


NatSci T 


.195 . ' 


.181 


.573 


-.294 


NatSci % 


.197 


.184 


.593 


-.305 


Phys T 


• .133 


.187 


.591 


-.365 


Phys % 


.138 


.187 


.712 


—•363 » 


Arts T 


.157 


.165 


.633 


-.296 


Arts % 


.,154 


.166 


.836 


-.281 



in the printing process and had'tb be discarded. Such profiles were not 
discovered until they had been given to subjects. And third, occasionally 
one or more profiles were inadvertently skipped by a subject and w^re not 
noticed until they were scored. These occurrences meant a reduction in the 
number of profiles which could be used to calculate the correlations. 

Correlations were calculated from both sets of trait scores on the pro- 
files. There are minor differences between the two sets of correlations. 
The correlations computed from the percentile scores were used in subse- 
quent analyses. This was done not only because absolute values were slight- 
ly higher but also because rtost subjects seemed to make their evaluations 
by using the percentiles. These scores appeared on the profiles as numbers, 
rather than as bar graphs like the T-scores. Observations of subjects' be- 
havior and -markings left on the profile printouts indicated evaluations 
based on the percentiles rather than, the T»scores. The concept of a per- 
centile seemed easier for subjects to understand than that of a deviation- 
based T-score. 

The means for the correlations as they appear in the tabJ-es are arith- 
metic means and are not computed by the z-score transformation. There may 
be arguments as to whether they represent unbiased measures of the popula- 
tion correlations. The a- trans format ion would give a slightly higlier esti- 
mate of these population values. 




As there was 'with the token weights » so here thet^ is a great deal of 
disagreement among subjects about how important each of the traits is. For 
^ almost every trait-*sex combination ther^ is at* least one correlation which 
'has a strongly negative minimum vaJ.ue« And the. maximum positive values in* 
dicate that at least one suJdject seemed to make overall judgments baseo al-- 
most, entirely on> the variation of one trait. » 

Although the correlation measures do not^ strictly speaking, form .an 
ipsative measure t there are some limits on their values. If the scores for 
the traits' were uncorrelated, then the squared, multiple 'correlation coef- 
ficient between the trait scores and subjectsV overall judgment woulo be 
equal to the sum of the squares of ^thc seven simple correlations between 
overall judgment and the trait scores (Darlington, 1968, p. 162^). Since 
the squared multiple correlation must be less than or equal to 1.0, it' was 
expected that the sum of the squares of the seven correlations would also 
approach 1.0 as a limit. 

For many subjects, however, the sum of the squared correlatipns x«fs 
either greater than 1.0 or far less than 1.0. The first case w^quld Indi- 
cate that the randomly generated trait scores were not free of random in- 
tercorrelations. The second would show that there were many subjects "^ho ^ 
were "baa" judges, who either did not make thei'.-^ judgments accordijf^g to a • 
consistent scheme or made a great number of errors in their calcinations, 
so that as a result tfte relationship of their judgments to trait scores is 
.only slightly higher than would be expected by chance alone. 

Many of the same relationships which appearea in the token weightings, 
appear with the latent measures as well. But in the correlations the r^ . • 
must be a lot of measurement error in the results, since oyer the whole 
sample the variance in subjects* evaluations explained by variance in trait 
scores is less than .56. This is calculated by summing the^-^quares of the 
seven correlation coefficients. 

In spite of the error there are observable differences in the means 
for each trait. These differenciss may be due to a number of "good" judges, 
judges who made a set of evaluations strongly related to one or two traits. 
A subject who tried to make only "minor distinctions in trait value or con- 
siderea the traits of about equal value would tend to have low correlatiqns 
for all the traits and be called a "bad" subject. ' ' . ^ 

The traits do seem to receive different values from the subjects de- 
spite great differences in opinion between subjects. ,A summary table pre- 
pareu oy stratum (Table 10) showsxjsome possible differences in trait value ^ 
for sex. 

9 

An analysis of variance was performed on the correlation method weight- 
ings, those computed from percentile scores, to test for significance in 
the differences observed. Prior to the analysis the correlations were 
transformed to 2-scores. Tne summary correlations presented in Table 10 
are computed from these z-score transformations. The same three-factor 
repeatea measures design from Winer (1962) was usea for this analysis. 
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TABLS 10 



— -iORREUTION TOKEN. WEIGHTS 
^, BY TRAIT, SEX, AND STRATUM 



Professionals Non* Professionals 
(N = 52) (N = 49) 



Trait 

Verbal 
Personality 
Quantitative 
.Social Studies 
Natural Science 
Physical 
Arts 



Bojr 


Girl 


Bojj. 


Girl 


.43 


.51 


.43 


.43 


.37" 


" . 35 


.35 


.37 


.29 


.27 


.33 " • 


.29 


.27-- 


.28 


.24 


.20 


.18 


.21 


.25 


.21 


.15 


.15 


.13 


.15 


.12 


.15 


. .17 


.22 



The results of the analysis of variance are presented in Table 11. 
The only significant effect is .for differences between the traits, which 
here were again significant at well beyond the .^001 level of significance. 
So in spite of all the,measurement error involved in the instrument the dif- 
ferences in the values'assigned by subjects to the traits were significant. 
However, the sex by-trait-interaction which appeared on the token method was 
not present here. 

There were a number of. other effects possible here which were not pos- 
sible in the analysis of the token method. A significant effect for stratum 
would have meant that either professionals or lay- perople were able to make 
"better" judgments on the profiles , judgments which resulted in a larger 
sum of the correlation coefficients. There were seme differences for 
strata, but not significant ones. If there were "better" judges, they did 
not belong to one group more than the other. In the same manner, it would 
have been possible for subjects to make "better" judgments on profiles of 
one sex rather than the other. But any such differences were not signifi- • 
cant. 

The only significant differences which did appear were between the 
trait values. Subjects did not seem to distinguish between the sexes when 
they made their evaluations, only between the traits, it could be that in 
a more lifelike situation the directly measured sex bias does not appear. 
It could be also that any such bias was swamped by the errors judgment 
which were made on the profiles. Or it could be that subjects were not 
made as aware of hypothetical student sex as they might have been. As can 
be seen from Appendix B, sex was indicated near\the top of the page while 
the bulk of a subject's attention was focused on "the scores^ and the judg- 
irent line trweh farther down the page. So it is possible that subjects did 
not pay much attention and thus did not notice when the sex of the profile 
changed,, • . _ 



TABLE 11 

ANALYSIS OF VARIANCE— CORR£LATION WEIGHTS 









nean ac^uw^e 


r 


Between Subjects 


• 6.729 


100 






w wX^a lUUi 


%\JOO / 


X 


• Uob / 


• b7o 






QQ 






Withii; Subject 


% 

lk7,3i+3 


1313 








* 059 


1 


• 059 


1 e .*i 


QuPaLUzn X bex 




1 




. 580 


Sex X Subject within 


3.6616 


99 


.0369 ' 














Trait 


17.77'> 


6 


2.9623 


24.727' 


Stratum x Trait ' 


.5593 


6 


.0932 


.778 


1 Trait x Subject within 


71.1867 


594 


.1198 




^ gBOUp 










Sex X Trait 


.258 


6 


.043 


.761 


"St rant um x Sex x Trait 


.2114 


6 


.0352 


.623 


Sex X Trait x Subject 


33.6116 


594 


' .0565 




within group ^ 











vi« p < .001 



✓ However, subjects did tnake distinctions between the importance of the 
traits by this inethod. They also made significant distinctions on the di- 
rect method* The question is whether the latent weightings derived from 
the profile evaluations are comparable to the deliberate weightings derived 
from the token assignments* 



Comparison of Methods 

It was hoped that the methods would be two aspects of the same general 
behavior and that the judgments made about trait value on one method woulu 
be comparable to the judgments made on the other* Therefore, the expectar 
tion was for general agreement between the two methods, within the limits 
of sampling and instrument error. / 

Tne first attempt to measure this relationship was to correlate the 
number of tokens a subject gave to a particular sex-trait combination with 
the correlation coefficient between his judgments and the percentile scores 
for the same trait and sex combination. Table 12 contains a summary of the 
results of this first analysis, Pearson product-moment correlation coef- 
ficients were computed between subjects^ token weightings and their 
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TABLE 12 



CORRELATIONS BETWEEN DIRECT AND LATENT WEIGHTS 
' FOR 101 JUDGES) 



Trait 


« 


Boys 


Girls 


Verbal 




.365 


.488 


Personality 




.578 


.474 


Quantitative 




.U52 


.343 


Social Studies 




.ii05 


.210 


Natural Science 




.358 


.155 


Physical 




.384 


.068 


Arts 




.169 


.331 




Mean s 


.387 


.236 



correlation weightings for the 14 trait-sex combinations. In reference to 
previous tables these correlations are between the first 14 variables in 
Table 3 and the percentiles correlations from Table 7 and 8. 

As the table shows, the relationships here, although different from 
zero, are not very high. They are a little higher for boys than for girls; 
the judgments of the subjects seem more stable across methods for boys. 
However, the variance in either set of weights accounts for less than 20 
per cent of the variance in the other set. 

There could be a number of reasons for this low reliability. There was 
certainly some measurement error in both methods. It is possible that sub- 
jects did not evaluate, students according-^ to their manifest values. Per« 
haps subjects made different kinds of judgments on the two methods. Or it 
is possible that they were not able or willing to use all the information 
available, seven trait scores and a sex designation, to make their evalua- 
tions. * 

The indication from these corrdations is that the less important 
traits are more subject to error, relative to their manifest judged impor- 
tance, when profiles are evaluated. The higher correlation coefficients 
are associated with the less important traits. Rank-order correlations 
were computed between the trait token means and the reliability coeffi- 
cients according to the ranks appearing in Table 13. 

These rank-order correlations suggest that subjects based, their judg- 
ments on only a few important variables rather than on all available infor- 
mation. For the greater the importance of the trait, the higl^er the relia- 
bili'ty was across measures. The judgments made on the profiles are not so 
concise about less important traits as about the moi'e importaiit ones, 

A second comparison of methods was made by computing Spearman rank- 
order correlation coefficients and Pearson product -moment correlation 
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TABLS 13 

RANKoORDERED TRAITS FOR 
RELIABILITY AND TOKEN-HEIGHTS 



Trait 



"Boys" 





Rel. 


Verbal 


5 


Personality 


1 


Quantitative 


2 


Social Studies 


3 


Natural Science 


6 


Physical 




Arts 


7 







= .572 



" Girls" 
Rel.' Token 



1 

2 

3 
5 
6 
7 



r7 
= .893 



1 
2 

3 
6 
•7 



coefficients between each subject *s vector of token weights and his vector 
of correlation weights. These coefficients were computed for the three sets 
of weights: for "boys", for "girls," and for the two combined. The means 
for these coefficients for the 101 subjects appieax' in Table 14. They were 
computed according to the z-score transformation. 



TABLE Ik 

MEANS OF RELIABILITY COEFFICIENTS (N -• 101) 





Spearman Rho 


Variance of 


Stan. Dev. 


of 






z-score 


J z-score 




"Boys" 


.554 


.264 


.514 




"Girls" 


.453 


.363 


.603 




Combined 


.607 


.255 


.505 






Pearson R 


Variance of 


Stan. "Dev. 


of 






z-score 


z-score 










. \ 




"Boys" 


.606 


.295 


.54% 




"Girls" 


.498 


.374 


.612 




Combined 


.679 


.304 


.,551 





This second a^^ysis also indicated more agreement between methods on 
trait importance for "boys" than for "girls." The combination of the two 
n^asures shows more agreement than for either sex individually. This may 
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indicate that sex differences were not so important as subjects jihought when 
they assigned token weights ^ that subjects were not aware of sex differences 
in their prdfile evaluations ^ or simply that more reliable measures are asso 
ciated with increases in number of measurements. 

Two correlations were computeA^ but the rank order correlations may be 
psychologically more me^ingfuL^ Wjects may be attempting to assign pri-* 
orities to the traits rather than attempting to assign ratio measures. Al«- 
though these data indicate somewhat more reliability and consistency across 
the two methods than the previous analysis, the variance in the weights as- 
signed to traits by one method accounts for less than ^0 per cent of the 
variance in the weights £vom the other. For some reason the average subject 
did not transfer value schemes exactly from one method to the other. 

The bentee> however, was formulated not as a measure of individual val«* 
ues but of group and societal values. A scientific analysis of an educa-> 
tional system would depend on system-wide values and goals, cultural and 
not individual stereotypes. So the relationship across methods of popula- 
tion and subgroup value profiles is very important. 

The traits were rank':u according to their judged values for the whole 
samplo. The rank-ordered traits appear in Table 15. The rank-order coef- 
ficients ore calculated for "boys," "girls," and both groups combined. 



TABLE 15 



RANK-ORDERED TRAITS 
FOR TWO METHODS (IJDl JUDGES) 





"I 


Sirls" 


Combined 


Token Corr. 


Token 


Corr. 


Token 


Corr. 


Verbal 1 


1 


1 


1 


1 


Personality 2 2 


2 


2 


2 


2 


Quantitative 3 3 


4 


3 


3 


3 


Social Studies 4 k 


3 


4 


4 


4 


Natural Sciences 5 5 


6 


5 


5 


5 


Physical 6 7 


7 


7 


' 6 


7 


Arts 7 ' 6 


5 


6 


7 


6 


r .964 


r - 


.929 


r = 


.964 



For these group values px^ofiles, there is a great deal of consistency 
across methods. The coefficients are all p?eater than .90 and indicate 
that for group responses the two methods result in quite comparable mea- 
sures • 
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Profiles of subgroups were also compared (across the two methods. Cor- 
relations are reported in Table 16, Relationships are still very high al- 
though slightly less than for the whole sample. Tnere is more agreement, 
nonetheless, across methods about "boys" than about "girls." Here, however 



TABLE 16 



CORRELATIONS BETWEEN RANK-ORDERED TRAITS 
FOR IDENTIFIABLE SUBGROUPS 



4 


, Professional 
(N = '52) 


(N = H9) 


Male 
(N = 67) 


Female 
(N = 34) 


"Boys" 


^ 1.000 " 


.929 


.96U 


.964 


"Girls" 


.786 


.838 


.893 


.964 


Combined 




.929 


.929 


.964 



the combined measures are slightly less consistent than those for boys, 
probably because there is extremely high consistency in group judgments 
about "boys." 

For groups and the whole sample, the agreement was much higher than 
for the average subject. There were many "bad" subjects, subjects whose 
profile evaluations showed no relationship to their token 'weightings and 
in some instances showed ev^n invers^ relationships • In group analyses 
these subjects tend to cancp^l each other out, although they do reduce the 
reliability of the irieasure and introduce error which clouds other possible 
relationships. 

Aa'Sffialysis' of many individual profile rating results discovered that 
the cc^rreiations between less important traits and overall evaluation did 
not exhibit more than chance variation from zero. There is, then, more 
evidence that subjects made their profile evaluations on some subset of 
all the information available, per^iaps just the more important traits. 

Yet group reliability analyses seem to indicate that there is concor- 
dance of opinion about the seven traits, especially for identifiable groups. 
It is imperative in developing a standard of educational benefit to be able 
to measure group values reliably and it would seem that either method can 
be suitably employed to obtain them. In fact, the direct method may prove 
to be the only one necessary, arid it is certainly easier to administer. It 
showed an expected sex bias which did not appear in the correlation method. 
Perhaps refinements might' be possible in the profiles to reduce some of the 
erx'or which seems to be present. » 
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D. Personal Variables 

Information was collected on stem J variables about the subjects and the 
interview schedule which might explain some of the variance which has been 
considered error of measurement. At the beginning of each interview session, 
subjects completed identification forms with information on residence, age, 
sex, education, and occupation. The interviewer also made, note of the sub- 
ject 'ts ethnic grouping. Fr^m these data, six background variables were con- 
structed. V 

\ \ ■ . 

The firsV variable is Location.' This variable also swves ad an iden- 
tification code for the. interviewer V'ince, as bt»en mentioned, data were 
also collected in the Cleveland and Connecticut areat' by two different re- 
searchers. Thus interviewer and location effects are inextricably con- 
founded. Subjects in the study recruited from Connecticut were coded 1, 
and those from tha Greater Cleveland, Ohio, area were coded 2. There were 
51 subjects from Cpnnecticut and 50 from Cleveland. 

» ( 

The variable Sex is a coding of the sex indicated on the identification 
forms. Male subjects were coded 1 and female subjects 2. Approximately 
two- thirds of the sample were males. 

The variable Stratum is a nominal variable, a categorization of the 
occupations of the subjects. Categories one through three contain the pro- 
fessional subjects; the first being high school teachers, the second high 
.school administrators, and the third teacher-educators. Categories four, 
through six contain the non-professional or lay subjects; the fourth can- 
prising all women regardless of occupation, the fifth, blue collar males, 
and the sixth, white-collar males. Table 1 (in Chapter 3) contains a con- 
plete breakdown of the sample by Lo'fcation, Sex and Stratum. 

The variable Age is simply the age in years as reported on the identi-. 
fication form. One female subject did not respond to this question and was 
assig^ied the mean age for the sample. Although the sample exhibits " wide 
distriisution on age, the mean indicates that more of the subjects come from 
the younger end of the curve. This does not appear to be a randan occur- 
rence but seem* indicative of the fact that older people were more afraid 
of participating in the study. 

The variable Education is the' completed grade level as reported by the 
subject. A high school diploma was coded 12 and a college degree, 16,, A 
Tnaster*s degree was considered one year's schooling beyond a college degree 
and coded 17, and a doctorate was considered two years' more work and coded 
19. Teachers and administrators who indicated having received a sixth-year 
professional certificate were scored 18 on this variable. ' Table 17 contains 
a description of variables Age and Education. 

It may be noted that the mean education is equal to almost a college 
degree even though post-graduate degrees are scored rather conservatively. 
Years of schooling for the sample would be expected to be a little higher 
than for the general population since all the professional subjects would 
have 'earned at least a college degree. But the non-professional group 
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TABLE 17 
AGE AND EDUCATION OF SUBJECTS 





Mean 


Stan. Dev. 


Maximum 


', Minimum 


Age 


36.08 


10.98 


67 


18 


Education 


15.64 ; 




19 


8 



also tended tQ somewhat well-*educated| probably a characteristic of the 
type of people 'willing to ta^e part, in the study. Les^s well-educated sub- 
jects were afraid to make judgments about educational values i pleading lack 
of proper credentials suitable intellig^ce. There were a couple of po- 
tential subjects who e ^n listened to an explanation of the task-^ and then 
abstained from being subjects i claiming that they were net- qualified to make 
such evaluations. ^ 

The variable Race contains the ethnic background of the subjects as re- 
corded by the interviewer. Since the sample was prednminently white , only . 
two categories were constructed} whites (coded 1) , and non-whites (coded 2). 
Only six of the 101 subjects were not white » of whom five were Black t and 
one Puerto Rican. 

Two variables recorded the order in which the materials were presented 
to the subjects. A tabulation of these two variables appears in Table 16. 



TABLE 18 
INTERVIEW MANIPULATIONS 

" Boys< * First. " Girls" First Total 

Tokens Before 32 ' lU 46 

Tokens After 22 30 52 
Unrecorded 3—3 

TOTAL 57 ' 101 



Subjects were asked to make their token assignments before or after 
they had evaluated their set of profiles. This manipulation was designated 
prior to each interview session. If a number of interviews were conducted 
consecutively one subject would assign tokens before evaluating profiles ^ 
the next the reverse. However, since no -recoz^d of the ord^'^r was kept from 
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one interview ses.sion to another*, the odd-even split was not syst^atic 
throughout the entire schedule of interviews. The unsystematic assignment 
did-result in an almost 50/50 split although a few more subjects assigned 
tokens after they had rated profiles than before. 

Subjects' profile packets could contain either "boys" or "girls" first. 
This random order was generated by the computer program when it produced the 
profiles. The token assignment was performed in this same compjuter-generated 
order, assignments being made for "boys" or "girls" first accoriding to the 
order of the profiles. More packets contained "boy" profiles first. 

These eight variabiles— six per-sonal, and two experimental— were corre- 
lated with the m token weights and 21 correlation coefficients, between 
subject rating and trait T-score for the "beys" profiles, /or the "girls" 
profiles, and for the /6«nbined set. Table 19 contains all significant inter* 
correlations'. Since there wez^ no significant correlations for, the inter- 
view manipulations they are not included in the tables. . . 

TABLE 19 

COliRELATION MATRIX FOR PERSONAL VARIABLES 





Location 




Sex 


Race 


Educat ion 


Stratum 


Ld\cation 








ft 




Age >v 














^ Sex 








.25 






Race 


n 




.26 




-.19 




Education 








-.39 




-.15 


Stratum 










-.45, 

1 





The intercorrelations between the personalyvariabies all seem to be 
idiosyncracies of the^^ample or of the definition of the variables. More 
minority subjects were Homen. The one Puerto Rican subject , who was the 
only subject who Kad not attended high school > is partly responsible for 
the race-education correlation since there were only six minority group mem- 
bers. The very strong correlation between stratum and education shows that 
the professional educators were better educated than the non«-professional 
edvicators. 

Significant correlations between the personal variables and token 
weights appear in Table 20* According to correlations » subjects from the 
Cleveland area gave more tokens "^o Quantitative and Natural Science than 
subjects from Connecticut > who gave more toker^ii^ to Personality and Physical. 
This relation may be due to an interviewer effect , as has been pointed out* 
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TABLE 20 . ' 

CORRELATIONS BETWEEN PERSONAL VARIABLES 
AND TOKEN WEIGHTS 



Location Age Sex Race Education Stratum 



Quantitative "Boys" 
Quantitative "Girls" 
Arts "Boys" 
Arts "Girls" 
Physical "Boys" 
Physical "Girls" 
Personality "Boys" 
Personality "Girls" 
Natural Science "Boys" 
Natural Science "Girls" 



.32 
.27 



-.28 
-.30 
-.32 
-.27 
.21 
.21 



.27 

-.23 
-.21 



.31 



The only other significant relationships occur for Education. Better 
educated subjects thought Arts more important for boys, and Natural Science 
more important for girls. They considered Physical less important overall 
than did less well-educated subjects. 

Finally I the significant relationships between the personal variable 
aiid the correlation weights appear in Table 21. 

Subjects from Cleveland i "ill rated Quantitative more important for 
all students and Personality less important for girls than subjects from 
Connecticut did. But their biases against Physical and for Nati^al Science 
measured on the token method, did not carry over to the latent measurement. 
The correlation between Location and Personality for the "boys" was not 
quite significant. 



The relationship of Education also changed. On the correlation method, 
better-educated si^jects gave higher weights to Verbal, a different rela- 
tionship from those which occurred on the token method. Three other re- 
lationships occurred. Older subjects rated Natural Science higher than 
younger subjects; possibly the younger generation is not committed to a 
traditional curriculum and thinks the hard academic subjects are not so 
important. And man rated Arts more important for boys and Personality more 
important for girls; the higher rating for l^ersonality may indicate some 
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TABLE 21 

CORRSLATIONS BETWEEN PERSONAL VARIABLES 
AND PROFILE WEIGHTS 



Location Age Sex Race Education Stratum 



Quantitative "Boys" .27 
Quantitative "Girls" .28 
Quantitative Combined .31 
Arts "Boys" 
Arts "Girls" 
Social Studies ".Qirls" 
Verbal "Girls" 
Verbal Combined 

Personality "Girls" -.20 
Personality Combined -.21 
Nat. Sci. "Boys" 
Nat. Sci. Combined 



-.2i+ 



-.27 
-.20 



.22 
.24 



.27 
.26 



-.22 



latent "chauvinist" feelings in the male subjects. The correlation between 
Stratum and Verbal may indicate that educators thought it somewhat more im- 
portant than non- educators did. 

There was one other relationship which was discovered. Correlations 
were computed between the Sex of the profiles and the overall evaluations. 
A description of this variable is in Table 22. It appears to be randomly 



^ABLE 22 



'PROFILE SEX WITH RATINGS 



Sex 



Mean 
-0.023 



S,D,. 

0.189 



Maximum 



0.U81* 




ao 



r 



dis^ributoifl^^ound a mean of zero. However, this variable received a corre«> 
lation of ^^with the order of presentation of sex, within the profile set. 
This value indiclites that in general the sex which came first was giveji 
higher evaluations, i.e. that su^iects tended to give higher evaluation to 
thtt first half of their .set of pi vfiles. • ' . 



E . _ Factor Analysi s 

One remaining question has to do with whether subjects responded to, 
or valued, traits- individually or whether thdy responded t«) groups of traits. 
Since both sets of value variables have many of the attribut^es of ipsative 
measures, by giving more weight to a particular trait, a subject must take 
some weight or emphasis away from some other trait. 

In the analysis of the relationship of personal variables, to ,token\ 
trait values, for example, the subjects from Cleveland were shown to valiM 
Quantitative and Natural Science higher and Physical and Pers^onality lower 
than subjects from Connecticut.- If such patterns occurred in most subjects, 
it might be more meaningful to talk abWt factors rather than individual 
traits. 

A principal-components factor analysis was used to fij^d put how the 
data might be meaningfully summarized. An analysis was first performed on' 
the weights derived from the direct method of token assignments. The weights 
used were combined weights for both sexes'. . The intercorrelations of these 
weights appear in Table 23, The rather large proportion of negative corre- 
lations is due to the fact that the weights form an ips at iv^ measure (Cle- 
mans , 1965). / . " 



TABLE 23 
CORRELATION MATRIX FOR 





FOR TOKEN 


WEIGHTS (N 


= 101) 










Ver. Per. 


Quan. 


SocStu. 


Nat.Sci. 


Phys. 


Arts 


Verbal 


-.20 


. -.03 ' 


-.40 


. -.25 


-.37 


-.45 


Personality 






-.23 


-.54 




-.20 


Quantitative 






.03 


.34 


-.30 


-.27 


Social Studies 








.33 


-.23 


.12 


Naicural Science 










-.20 


.17 


Physical 












.27 



Arts 



\ 



Three factors emerged from the analysis. A standard varimax rotation 
of the thpee factors resulted in the factor loading matrix whi^h appears in 
Table 24, An oblique rotation produced substantially the -saire \f actor load- 
ing matrix. The factors were also shown by the oblique rotation to be al- 
most completely orthogonal. The largest intercorrelat ion between factors 
was -.14. , •■ I 



TABLE 24 

« 



FACTOR LOADINGS FOR TOKEN WEIGHTS 





1 


Factors 

— r 

2 


3 


1 

% 


Verbal 


.10 

■ 


-.55 


-.69' ■ 




Personality 


-.91 


-.12 


.07 




Quantitative 


.64 

1 


-.34 


.01 




Social Studies 


.15 


-.12 


.87^. 




Natural Science 


.74 


.04 ■ 


.43 




Physical 


-.20 


.81 


-.29 




Arts 


.13 


/.77 

1, 


.26 





/ 

The same principal-components factor Analysis was also applied to the 
trait weightings aerived from the correlation method. The weights used for 
the analysis were the correlations computed over subject's entire set of 
profiles, the combined weights for both sexes. The intercorrelat xon matrix 
for these weights appears in Table 25. Since these weights have some prop- 
erties of ipsative measures, a high proportion of negative correlations ap- 
pear in the table. 

Again, three factors emerged frcsn tne analysis. A varimax pvidtion^ 
of these three factors produced the factor loading matrix which appears in , 
Table 26. Further oblique rctatlon diu not change the loadings substanti- 
ally and showed that the factors were substantially orthogonal. Again, the 
intercorrelationons of the factors were not much different from zero, the 
largest being -.10. 

Clemans (1965)^ in his examination of ipsative variables, says that 
"it wilJ be extremely difficult, if not impossible, to oijtain psychologi- 
cally meaningful results from the factor analysis of a set of ipsative in- 
tercorrelations" (p. 51). However, in this analysis at least two meaning- 
ful factors did appear. The third factor was not easily identified and 
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TABLE 25 

CORKELATION MATRIX 
FOR CORRELATION WEIGHTS (N =101)- 





Ver, Per. Quan. Soc*Stu. 


Nat.Sci. •• 


Phys.. 


Arts 


Verbal 


-.16 -.OU -.17 


.OU 


-."07 


-.11 


Personality 


-.52 -.10 


-.2U 


.OU 


.02 


Quantitative 


.03 


.3U 


-.01 


-.08 


Social Studies 




.15 


-.06 


.06 


Natural Science 






.06 


.lU 


Physical 


• 








Arts 


f' 

/' 

/ 









/ 

. TABLE 26 
FACTOR LOADINGS FOR CORRELATION .WEIGHTS 



r 



i. 




Factors 






1 


2 


3 


Verbal 


.15 


,' -.69 


-.20 


Personality 


-.bO 


.10 


.14 


Quantitative 


.82 


.00 


-.05 


Social Studies 


.17 


.77 


-.15 ^- 


Natui^al Science 


' .55 


.lU 


.30 


Physical 


.00 


-.19 


.76 


Arts 


. .00 


.27 


.64 



Tnese three -factors account for b9 per cent of tne variance in trait 
weightings. 
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named. It may be an artifact of the way the data were organized, i.e., as 
an ipsative measure. 

As may be not fed from- the factor loadings matrices, the three factors 
which emerge from each method are practically identical. The first factor 
is definitely a "scientific" factor. Quantitative and Natural Science load 
very highly on this factor, and Personality has a very high negative load- 
ing. This factor makes intuitive sense. Natural Science and Quantitative 
are seen as cold, unemotional, and technical calculation, devoid of the 
warm richness and humanity of Personality* Subjects seem to 'see Personality 
as not related to the factual subjects of the curriculum. This factor ac- 
counts for 31 per cent of the variance of the token measure, and 25 per cent 
of the variance on the correlation measure. 

The second factor for the token weights and third for the correlation 
weights has been nick-named the "ballet" factor, since it receives high 
positive loadings for Physical and Arts, for the token weights analysis, 
this factor may be considered simply a "non-textbook" factor, since it also 
has some negative loadings for Quantitative and Natural S-:ience. 

The third factor does not make as much intuitive sense as the other 
two. As has been noted, it may be just an "artifact of the situation. One 
would expect Social Studies and Verbal to be strongly related. It might be 
considered a "social concern" factor, associated with some idea that it is 
better to be socially aware than to read Shakespears. There might be .just 
such a factor operating in the correlation weights, but the situation is • 
more complicated for the' tokens. The positive loading for Natural Science 
does not make sense unless the factor becomes something of a "practical" 
arts factor, negatively related to ivory-tower intellectualism. But the 
two loadings foi^ Arts ana Physical do not make any sense to that breakdown. 
So this factor remains ' unnamed , and difficult to explain. 

The factor analysis indicates that subjects probably respond to groups 
of traits rather than to traits individually. -The "scientific": and the 
"ballet" factors make intuitive sense to most traditional breakdowns of the 
curriculum. Science and math programs are collected together, and the 
stereotype of the engineer is that of an individual with no personality. ^ 
Correspondingly, the physical education and aesthetic fields are often dis- 
claimed by tracitional academicians. These two are factors that one might 
expect to find in subjects' view of the academic curriculum. 

The third factor is not s cience factor; there is also no relation- 
ship between it and the first factor in the oblique transformation. Per- 
haps, therefore, it would best fit in the social concern or social aware- 
ness category. Its exact nature is hard to discern. 

A way to clarify these factors might be to compute factor scores for 
subjects and investigate the relationships of these factors to personal 
characteristics of the subjects. 

Summary and Conclusions . Subjects were selected from two strata, 
people professionally engaged in secondary education ana lay people from 
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outside the educational system ^ and from two locations, in ps^io and in 
Connecticut # Thex'e wex^ 101 subjects in the sample, 52 pr<^^essionals and 

H9 non-professionals • y 

// 

Subjects V values toward the s^ven traits were sampled by two methods: 
a) directly, by having them assign tokens to the trails; and b) indirectly, 
by having them evaluate random profiles of hypothetical students. Subjects^ 
, evaluations of profiles were correlated with .the prp^file trait scores to, 
produce the indirect measures of tVait value. 

Subjects* value judgments were analyzed by an analysis of variance. 
For the direct method there were significant differences between the empiri- 
cally derived values. There was also a signi^cant sex-by-trait interaction 
the values of the traits being slightly but significantly different for boys 
and girls* For the indirect method there werfe significant differences be- 
tween the trait values, but there was no significant sex-by-trait interact* 
tion. For neither method was there any significant differen<?e between^^edu- 
cator and lay groups. .^.^--^ 

For the total scimple', ana for identifiable sub-groups, the measurea T 
order of trait values was consistent across methods. For individual sub- 
jects the overall order among trait values was not so consistent across 
methods. And the actual measured values of the^^traits, the values given by 
individuals on each method, were even less consistent. The average relia- 
bility coefficient Wc^ about .35. 

Some significant relationships were discovered between personal vari-, 
ables and trait values, buc these relationships were not necessarily con- 
sistent for both methods. They also were not found to relate to the group- 
ings in the sampling design. 

A principal-components factor analysis indicated that subjects' trait 
values, fall into patterns which emerge as three factors of trait. The 
first factor was a "scientitic" factor, with high positive loadings for 
Quantitative and Natural Science ana a high negative loading for Personal- 
ity. The second factor was a "non-textbook" factor, with high positive 
loadings for Physical and Arts. The third factor was difficult to name. 
It had a high positive loading for Social Studies, a positive loading for 
Social ^tudi?iS, a positive loaaxng for Natural Science, and a high negative 
loading for Verbal. Very comparable factors emerged from both methoos. 

The study found that it was possible to discover differential weight- 
ings between subject areas, i.e., that measurable systematic differences do 
occur in the judged importance of academic tz^aits. These differences ex- 
hibited themselves whether subjects were asked to rate areas directly by 
a token assignment, or where asked to evaluate student profiles with scores 
on these areas. In both cases the order of importance of the traits was-^^ 
almost exactly the same for the total sample and for identifiable sub-groups 
v;ith ' nly slight differences among some of the 'tx^aits. 



It would thus seem possible to construct a measure like the bentee, a 
f;eneral jneasure of educational advancement • With vigorous sampling tech- 
niques one could, with relative certainty^ measure the educational values 
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of the concerned conmuiaty. Analysis has shown that the direct measurement 
of values produces as good an estimatiori of this population value profile as 
~tHe~lndii<gct-Tnethod-does, with -far-greater ease- of -administration* 
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CliAPTER 5 
SUMMARY AND CONCLUSIONS 



Introduction > The tqols of management science and operations research 
have not yet been applied to general educational problems. They have been 
utilized in specific areas where specific goals can be optimized, but they 
cannot be applied to optimum solutions until the overall goals of education 
are specified* Prompted by the notable lack of a well«»defined measurable 
goal for educational programs and research, we have proposed that such a 
measure be developed by empirical methods* 

Educators have long labored in a fog of underfihed goals. For want of 
a better measure, the effects of schooling and education have usually been 
measured in dollar terms, by income and social-class effects, or by stan- 
dard achievement tests. Although educators are in general agreen^eiit that 
the .goal of education is not purely to raise income levels or particular 
achievement levels, their published statements show little agreement about 
what that goal might be or how onev might measure progress toward it« 

The reseajTch tested a methodology for empirically defining the goals of 
education and measuring their relative importance. The success of this 
methodology would allow the development of a universally applicable measure 
of educational benefit. This measure, derfved empirically from sociil and 
cultural values, could be used to evaluate students and programs and could 
be defined in responise to social and cultural changes. It could serve as 
an effectiveness criterion for rigorous scientific systems analysis. 

Summary . The high school curriculum was categorized arbitrarily into 
seven traits: Quantitative, Arts, Physical, Social Studies, Verbal, Per- 
sonality, and Natural Sciences. These titles are summative of concise defi- 
nitions which were developed for each trait. Random student profiles were 
generated with scores for these traits. 

Two samples were selected, the first of 52 people professionally en- 
gaged in so^ aspect of secondary education, and the other of ^9 people not 
associated directly with education. Value judgments were elicited by two 
methods. Subjects were asked to weight the seven tratis by assigning them 
tokens. They, were also given a set of random student profiles and asked to 
rate each of these hypothetical students on a continuum^ 

Hypotheses were generated about the results of these two rating tech- 
niques. It was expected that some of the traits, possibly Quantitative ana 
Verbal, wnich are analogous to tne main cosnponents of many achievement tests 
would be rated significantly higher than others .i^ Other hypotheses were con«- 
cerned with possible systematic differences in judgea trait importance for 
boys and girls, with differences in trait- importance judgments between pro- 
fessionals and non-professionals, and with the relationship of individual 
differences in judged trait importance and the personal variables which were 
measured. The two methods were expected to be somewhat equivalent ana 
highly correlated* Finally, subjects were expected possibly to value trait 
clusters rather than individual traits so that trait factors might emerge. 
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The data were prepared and variables constructed for the analysis* 
Various, correlational analyses, analysis of variance » factor analysis, 
werejused to test the hypotheses. 

When judged by the total sample, the order of the traits in impbrtance 
was practically the same for both methods* For the boys this judged order 
for the token method was Verbal, Pexsonality, Quantitative, Social Studies, 
Natural Sciences, Physical, and Arts, with the qrder of the last two traits 
reversed for the profile-rating method. For the girls the order by the 
tokens was Verbal, Personality, Social Studies, Quantitative, Arts, Natural 
Science, and Physical, with Social Studies and Quantitative reversed, and 
Arts and Natural Sciences reversed, in the profile ratings. 

Two analyses of variance were performed, the first using the values 
from the token method and the second using the trait values from the pro- 
file-rating correlation method. Both analyses showed significant (p < #001) 
differences in trait importance. There was no significant difference be- 
tween the behavior of the professionals and the non-professionals on either 
method. Differences within these groups were greater than any identifiable 
group difference. For the token method there was a significant (p <' ,01) 
sex-by-trait interaction— some traits were judged more important for boys 
than for girls and vice-versa--but this interaction did not occur in the 
profile evaluation method. 

The study seems to have produced reliable measures acro^. met hods, when 
values were averaged for groups of subjects. The major difference (between 
the order derived from the token method and that from the correlation method) 
is that the significant sex difference in trait importance which occurred 
for the token-as3ignment method did not appear in the profile-rating strategy. 

There were some significant correlations between personal variables and 
judged trait importance. With such a large number of correlations, a few 
significant ones would be expected by chance alone, 'One of the relation- 
ships that did occur was for Location; subjects from Cleveland tended to 
give more importance to Quantitative and Natural Science and less to Physi- 
cal and Personality, This "regional" effect could be attributable to some 
experimenter effect, however, since the two samples were collected by dif- 
ferent researcliers. 

There were also significant relationships for education and age. More 
educated people judged Vex'^bal and Arts to be more important; older people 
gave greater importance to Natural Science, Finally, women tended to rate 
Personairty and Ax^ts of less importance for boys than did men. 

Co nclusions . This study hae shown that, even with the measurement 
erx^or"Tnherent in a newly developed and unfamiliar technique, it is pos- 
sible empirically to weigh various curriculax* ax-^eas in accordance with their 
importance for an overall n^easure of educational advancement and benefit. 
Thus the ar-^lication of operations research to the educational system is 
more than j^cit a hypothetical possibility. 

Seven "academic** traits were developed from a traditional breakdown of 
tae school cuxTicuium by logical analysis. By both latent and dix'ect 
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measurement schemes » these seven trait^^ere weighted by 101 subjects and 
received.weightsXwhlch were systematically different from each other. These 
trait weightings 'Trom each method, were comparable when averaged over groups 
of subjects* 

■. 

The direct method of weighting by* tokens evidenced some expected dif- 
ferences in trait importance by sex. These differences did not appear when 
values were sampled latently through judgments elicited on student profiles* 
There was a great deal of error associated with these judgments about hypo- 
thetical students, just as there well might be in unguided human judgment 
about actual people. 

The measurement error in the profile ratings may have been due lo sub- 
jects* unfamiliarity with the instrument or to difficulties in making such 
complex judgments with so many bits of data. Since the weightings derived 
from both methods were so comparable for groups , and the latent method in- 
volved so much error, it may be unnecessary to make more than a direct 
assessment of values. Such a method is easier to administer, and seems to 
supply at least as much information as a latent value assessment. 

Whatever the method, the value profiles, the vectors of trait weight- 
ings, were not statistically different ^or comparisons involving a group of 
professional educators and one of the non-professionals. Since the groups 
were not * sciiected randomly, it cannot be considered that no differences 
occur for income and occupational groups, but in this study such differences 
were swamped by differences between subjects within groups. 

A measure as important, potentially, as the bentee, should clearly be 
a group measure, and care must be taken thc^.t it represent the social group. 
For if students and schools are to be widely compared on this measure to 
facilitate scientific evaluation and development of programs, that measure 
must be derived from the society at large. 

:p Suggestions for Further Research . Some refinements are possible in 
the^ instrumentation, and in' the selection and definition of the curriculum 
areas or traits. The present study employed a traditional breakdown of the 
high school curriculum. Further effort may be applied to a development of 
the logical or latent structure of the curriculum. Some method, such as 
latent partition analysis (Wiley, 1967), may be profitably used and recur- 
sively, applied until a latent structure of the curriculum emerges. The 
techniques of this study may be applied in each level of such a structure 
until the point is approached where no meaningful differential weightings 
can be obtained by empirical methods. 

Of greater importance, though, is the. investigation of the interper- 
sonal difference^ which occur in this study. The sample for this study 
tended to be homogeneous, in some respects,, consisting generally of those 
who were comfortable enough with their educational system. It might be 
labeled a middle class, middle- income , white, youthful and v/ell-educated 
group; even blue-collar subjects tended to fall into the same classifica- 
tion. 
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What could be implemented is a more complex sampling design, one that ' 
might sample across class , income , and occupational groujps with a more ef- 
fective recruitment method, i.e., one that would influence more subjects to, 
take part in the study. Such a design would allow greater tonparisou of 
different groups and might discover some" of the factors which contributed 
to the large wi thin-group differences occurring in the judgments.. That 
there were no di-fferences- between, professionals and non-professionals may 
be due to the similar social class backgrounds of the /two samples. 

Finally, further study might be applied to comparing direct with latent 
methods for trait weightings. This study found a very strong relationship 
between them. With fewer traits the error involved in the profile weight- 
ing may be reduced. 

Howevert as is indicated by the correlation analyses, there is lew 
individual reliability for, the two measures, at least when there are only 
50 profiles. On the average, the variance in the number of tokens assigned 
to a trait accounts for barjely ten per cent of the variance in the measured 
relationship between profile rating and trait score. Since the relation- 
ship is weaker for less important traits, it may be partly a function of 
the number of trait scores a subject could comfortably consider when making 
his judgments. It might also suggest, however, that subjects made different 
judgments in their token assignments and profile ratings, which somehow . 
were cancelled out by the groups as a whole. 

Nevertheless, rank order correlation coefficients for the trait impor- 
tance order for each subject on the two methods has a mean greater than .50. 
This may indicate that differences were due more to measurement error, or 
to subjects' inability to weigh as many as seven important scores in makii^g 
an assessment of a student. 

A principal-components factor analysis performed on the seven o'^erall 
trait measures, weightings combined for both sexes, produced equivalent 
sets of three factors. The first was a "scientific" factor, with high load- 
ings for Quantitative and Natural Science and a negative loading for Per- 
sonality. The second has been named the "ballet" factor, with high posi- 
tive loadings for Physical and Arts. And the third, which is difficult to 
name, has a high positive loading for Social Studies and high^ negative 
loading for Verbal. These are all the loadings for the factors derived 
from the correlation methods. For the token method. Verbal loads nega- 
tively on the "ballet" factoi' and Natural Science loads positively on the 
third factor. Subjects' judgments do then, as supposed, fall into response 
sets. 

This study nas been but a first step toward estimating an empirically 
verifieible measure of educational values. The search must continue for a 
commonly held measure of educational advancement, a standard which can be 
used to measure the benefits of education for each member of the society, 
and for the society as a whole. 
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Initial Telephone Contact 



Hello. (May I speak to your mother or your father?) 

Hello, is this Mrs. ? My name is • 

I'm sorry to trouble you. I am a resear'ch associate at The Uno-versity of 
Connecticut where we are conducting a public-opinion survey to determine 
what people believe makes som?Lone a "well-educated" high school graduate.^ 
You ai^e one of the people I have been contacting in this area of (Connecti 
cut or Ohio) to see if they would be willing to participate in this survey 
Would you be willing to help us with your opinion? 

(No) I am sorry to have interrupted you. Thank you for your time. 

(Yes) Thank you. We have a number of response categc/ries in which we 
place subjects. To determine which of the resporse categories your house- 
hold might fall into, could you tei; me what your husband do.^s for a liv- 
ing? (White collar or blue collar occupation?) (At this point, determine 
which response category this household >ill be put^into - Housewife, Blue 
Collar Worker, or White Collar Worker.) , 

A. (IF HOUSEWIFE) .We have some forms which are profiles of "high school 
students on various activities, which we would like you to look at and 
evaluate. This should take about 30 minutes of your time. We have been 
authorized to pay°you,^$6.00. When would be a convenient time for me to 
come to your home to show you the material? ° 

\ 

(Make definite appointment) 

Thank you for your coopGration. I will see you (at appointment time ). 

B. (IF BLUE OR WHITE COLLAR WORKER) Do you think your husband would be 
willing to participate in the si^vey? We have some forms which ^e pro- 
files of high school students on various activities, which we would like 
him "to look at and evaluate. It should take about 30 minutes of his time. 
We have been authorised to pay him $5.00. (IF WIFE AGREES) When would be 
a convenient time fo.c me to come to your home to show your husband the 
materials? (Make definite appointment). Thank you for your cooperation. 

I will see you ^d your husband (at appointment time). 
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DESCRIPTIONS OF THE TRAITS . 

QUANTITATIVE 

The use of numbers in school and in life. This refers to arithmetic compu- 
tation and reasoning, algebra, geometry, statistics, trigonometry, and, for 
"the better students, perhaps beginning calculus and computer progreunming. 
It includes the ability to use math in applied problems, and to read graphs 
and tables. A high score would imply strong methematical achievement and 
advancement. A low score would mean low or liited success in numeric 
operations. 

ARTS 

A 

Activities, skills, and concepts usually considered esthetic: drawing, and 
painting, modeling clay, and knowledge of famous artistic work of others i 
musical performance, composition, and appreciation, performance in drama, ^ 
interpretative reading, and poetry; photography and f ilm-makingi- and multi- 
media presentations. A high score would imply ability in more fields, or 
perhaps distinction in one or more. A low score would mean a noticeable 
weakness in esthetic ability. 

PHYSICAL 

Persona Ji/ health and hygiene, strength, posture, stamina and agility, as 
well as performance in sports and gymnastics. Sports may include team 
sports such as -basketball, and individual sports such as tennis or bowling. 
Included also at^e such activities as camping and related skills. A high 
score would imply superior vitality and athletic ability. A low score may 
n«an some physical or sensory weakness. 



SOCIAL STUDIES 



\ 



Knowledge of history, culture, civics, social problems, anthropology, psy- 
chology, polities, government, and citizenship, so far as these can be 
expected to be masterea through the twelfth gi^ade. A high score would mean 
advanced knowledge of many of the fields associated with the political, 
social, and cultural behavior of the humag race. A low score would imply 
little knowledge of these facts and principles. 

VERBAL ■) 

The effective use of language skills i reading, writing, speaking, conversa- 
tion, prose, and poetry. The ability to write speeches and debates. Vo- 
cabulary anu verbal reasoning, h' high score would imply ability in many 
of these fields, probably ability to use formal grammar and rhetorical 
structure, and some ability in foreign language as well. A low score 
would mean lack of facility to express oneself cleai-iy and with acceptea 
usage, and poor reading ability. 
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PERSONALITY 



This refers to many social behaviors of the student » his habits, and chav^ 
acteristics of his relationships to other people, with some regard to the 
morality and propriety of those behaviors* Included ai?e social popularity, 
personal appearance, apparent attitudes toward those around him, ability to 
meet people and interact constructively, to develop strong and lasting 
friendships. A high score would imply definite strong personal leadership 
and character. A low score would imply a more withdrawn $ ineffective, or 
deviate personality • ' ^ * 

NATURAL SCIENCES 

Ability and achievement in the physical and biological sciences, within the 
expectations of twelfth«grade students^ Included are knowledge, skills, and 
concepts in general science, health, environment, biology, physics, chem- 
istry, astronomy, botany, physiology, zoology, and simila?^ sub jects, which 
might be learned by students of this age. A high score would mean high 
general learning in these fields, with perhaps distinction in one or more/ 
A lev score would imply ppor performance in these areas, a lack of under- 
standing of the principles or facts of science. 
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INSTRUCTIONS TO THE EVALUATORS 



As I mentioned, we are doing a type of public opinion survey; we are 

doing a survey of educational values. ,We are trying to find out what people 

mean when they use such terms as "well-educated" or' "weJLl- prepared". (We 
intend the terms here to be somewaht synonymous.) 

What might we mean when W2 say that a person- is "well-educated" or 
"well-prepared?" There are many wi^s of angering this question, of making 
\^uch a judgment,' depending on-, what qualities and abilities one might regard 
as important. We are asking you to help us decide, and to let your own 
values and beliefs be your guide. . 

' ' So we are asking you to judge fifty (50) twelfth-grade students, 25 
boys and 25 girls, to rate them for how "well-prepared" or "well-educated" 
they are. These boys and girls will be presented^ to you by means of a stu- 
dent "profile", such as this one, and all that , you will know about them is 
contained in the trait scores, percentiles and T-scores, on tne profile. 
(These students are not real, the profiles w^re madeup for the purpose of 
this study. ) 

el 

XS-ample profile presentedo) 

To help you understand what these profiles mean, here are some rough 
descriptions of each trait. \^ . ^ 

t 

(Descriptions of the traits are given to the evaluator in the ^order printed 
on the profile.) 

You should study these descriptions until you have a fairly solid 
understanding of what kinds of ^abilities are included in each one and of 
° how you feel about the import aj.v.e of each for an educated person. No iri>- 
portai:ee should be attached to tl.>;^ order of the traits. They are arranged . 
in a chance order: for each set of profiles the computer randomly rear- 
ranges the trait order. 

Since the descriptions of the traits may be somewhat arbitrary, please 
■ read them cai-efully and I will try to answer any quf':.tions you may have 
about them. 

(Answer questions.) 

Mow it is up to you to decide which of the *raits are more important 
for you. It is unlikely that you would consider that they are exactly 
equal in importance; some will probably be more im-.ortant than others. 
Please try to consider the exact importance you would give to Sdch one. 
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If you are readyr .then we \tilX continue. ^ 

A. (If tokens are first * otherwise, got to B.) 
(defwre we go back to the sample profile) please spread out the trait oards. 
Here arie 50 pennies. I would like you to assign pennies to the traits, by 
placing them on the card, accordi^ng to how important you believe each one is 
for, a twelfth-grade (boy or girl t* first one listed on the profiles). If 
you should decide that only one is really important, then you woiild put all 
the pennies on that card. If you think one or more of the traits'^have no 
importance, then you would put no penhies on these cards'* Most likely, how- 
ever, you will feel that most of the traits deserve some pennies* Remember, 
this tims you are weightinig them for a twelfth^grade (boy or girl, as above). 

J)o you have any questions? (Answer the questions.) 
Okey,j please put the pennies on the cards .1 Whien„yoiN^ satisfied with the 
way you have arranged them, I will record each number .^Tl^se numbers should 
add up to 50.' (Wait and record.) "-^^ 

Now do the same thing again for a twelfth-grade (boy or girl, opposite 
sex irom .above). When you are finished I will again record it. 

fi. Now you should ^go back to the profile of the twelfth-grade (girl 
or boy) which I gave you, to understand in what ways he/she is outstanding, 
average, or poor. To help you understand the numbers and the bar graphs 
used, here^ is a brief explanation of the profile. 

Ther^ are two fSCor2S for each trait. In the column headed %IIjE, you 
will find, the percentile score achieved by this student ^ compared with other 
twelfth^grade students, po you k^now what a percentile is? To refresh you, 
here is a brief explanation. 

In the sample profile the (boy /girl) has a percentile score of ^^^^.^^..^ 
for the trait . (Pick ofxt a central one.) This means that this 
partj.cular student scores better than % of the other twelfth-graders , 
and % score better than he does. Let us look at a higher one. (Pick 
out a hig^ one.) What does the a fter mean? It nieans that this 

student is very good indeed ^n , compared with other students, better 
than - % of them and only % 'are better than (he/she) is. And 
if we look at a low one, (Point out low one) like , we see that this 

student is only at the percentile, (he/she) is better than only % 
of the other twelfth->grade sti^dents, and % are better than (he/she) 

is^. And so on. ^ 

\ ■ 
* \ 

Do you have any questions about this part? (An^er questions.) 

Another way of describing the students is by the bar graphs on the 
right. Jhese represent T-scores, which are a transformation of the per- 
centile distribution to the normal curve. To understand these scores we 
draw a picture of the normal curve above them. You can see from the curve 
how these scores tend to bunch together toward the middle, as is indicated 



by the height of the curve, ^ in that area, and' how very few scores fali to- 
t^ari the ends. For the percentiles, however, .the same, number of. people get 
each score. The T-s core distribution shows that the one percentile point 
difference in score at ^he middle of the curve is'^not anywhere near as mean- 
injgful as a one-point difference would be at the ends. \ . 

^ As I said these scores are just translations of the percentile scores • 
to the normal curve, to, bunch the^ middle scores and spread out the- ends. 
The- scores range from about 20 to 80, with the extremes being very rare. 
The higher the percentile' score, the longer will be the bar* You should be 
able to tell by looking at the ;,bar, compared with' the curve above,! just 
about how strong or how weak the student is on any pat'ticular trait. ^ 

Remember that both the percentile score and the bar graph are just two 
ways of showing how the student compares with other students on the particu- 
lar trait in question. • . 

' -■ ' ' ' 

Do you have any further questions? (Answer questions.) . . , 

Now that you are familiar with the profile and the descriptions of the 
-traits, I would like to ask you to look again at the profile and to make 
some decision about how •♦well-prepared" or "well-educated" you think' this 
student is. •> 

• * i 

If you value the traits in which (he/she) is high and don't think the 
low ones are very important, you may decide that (he/she) deserves a high 
score, perhaps an 8, 9, or even a 10. Then you should make a mark at the 
point where you want it to count along the line marked YOUR OVERALL JUfiG- 
MENT which appears below the, prof ilip. 

On the other hand, if you regard the low traits as most important, and 
the others less so, then you may want to give (him/her) a low score, and 
make a mark in the 1, 2, or 3. Or you might value the middle traits, in 
which case you might give him a middle score. Decide now hew you feel, 
and make a mark , ^ 

The easiest mark for me to interpret would be a vertical line at the 
point you desire. Remember that there are more than 10 points on the scale. 

Any questions? (Answer questions), 

Now I will give you the 50 student profiles which will look the same 
as the one we have been discussing, except that the scores for the traits 
.will be different. Also, the first 25 will be (boys/girls) like this one, 
and the second 25 will be (girls/boys). On each one please make the same 
kind of judgment you just made on the sample, end make your mark somewhere 
along the line. Remember that there are no "right" or "wrong" answers. The 
only right answer is the one which agrees with your own best judgment about 
the student presented in the profile,. 
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• 'Any last questions? (Ai»sWiW questions.) ' 

Are you ready to begin? Then here ajce the profiles, tty to make judg- 
irte'nts at various points along the line, I will be back when you awe finished 
so I can collect thi) evaluations and give you your check. , 

(If the tokens come after, go to A.). I will have another short thing 
for you to do then. (After, for the tokens, the ins-tructions will be the 
same as above, excepc that "before we go back to the sample profile" will 
be deleted. Those three paragraphs will be^^ed. 

(When the profiles are finished, they will be collected and the evalu- 
ator will be given his stipend. The researcher may discuss the project with 
the subject, if there is time anrl interest,) 



